MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Model Compression
Keywords: Low-Rank Pre-training, Rank Budget Allocation, Magnitude-Direction Decomposition, Optimizer State Compression, Gradient Projection
TL;DR¶
By identifying the structural sensitivity disparity across Transformer linear modules and mitigating the joint attenuation of gradient magnitude and direction, MoARa introduces static profiling-based rank reallocation alongside head-dimension-aligned block-wise decomposition, achieving 37% fewer optimization steps and 34% wall-clock pre-training speedup on Llama 2 7B without expanding the total rank budget.
Background & Motivation¶
The computational and memory footprints of large language models (LLMs) scale sharply with parameter count, context window length, and training token budgets, positioning optimizer-state memory overhead as one of the most critical systems bottlenecks in large-scale pre-training. Under conventional full-parameter AdamW optimization, tracking first- and second-moment statistics requires maintaining two persistent tensors of identical dimension to each model weight matrix, effectively tripling or quadrupling the active memory requirement beyond raw model weights and activations. While low-rank gradient projection frameworks such as GaLore have emerged as a promising paradigm to compress optimizer memory by projecting full gradients into a low-dimensional subspace via periodic truncated singular value decomposition (SVD), these methods suffer from severe optimization inefficiency: reaching a target validation perplexity typically demands prohibitively prolonged training iterations and wall-clock schedules.
A rigorous investigation into this convergence bottleneck reveals two fundamental algorithmic design flaws in existing low-rank pre-training schemes. First, current frameworks distribute the projection-rank budget uniformly across all Transformer projection weights (specifically the seven attention and MLP matrices: attn.q/k/v/o and mlp.up/gate/down). This uniform allocation implicitly posits that all functional modules possess identical sensitivity to subspace projection, ignoring the sharp empirical heterogeneity where token-routing projections and representation-expansion bottlenecks exhibit vastly different alignment tolerances. Consequently, uniform allocation leads to an inefficient state where redundant modules are over-provisioned with rank while critical capacity bottlenecks are starved of representational degrees of freedom. Second, directly projecting raw gradient matrices compresses local gradient magnitude and spatial direction jointly within the truncated subspace, eliminating fine-grained scale variation across channel blocks and attenuating low-energy yet directionally vital gradient signals.
To overcome these dual structural constraints without enlarging the total memory budget, this paper investigates the information geometry of low-rank gradient projection and formulates an architecture-informed optimization framework. Core idea: establish a static, momentum-preserving module-aware rank allocation guided by baseline gradient cosine alignment diagnostics, coupled with block-wise magnitude–direction decomposition centered around attention head dimensions, isolating local scale tracking outside low-rank projection to accelerate low-rank pre-training convergence.
Method¶
Overall Architecture¶
MoARa is structured into a one-time offline profiling phase and an active dual-branch low-rank pre-training pipeline. During initialization, a brief 1k-step exploratory run profiles the cosine alignment score \(S_\tau\) between full gradients and their low-rank reconstructions across Transformer module types; combined with single-module rank sensitivity ablations, this identifies the bottleneck receiver set \(\mathcal{R}\) (mlp.down) and the robust donor set \(\mathcal{D}\) (attn.q, attn.k), producing a static rank reallocation that strictly preserves the total rank budget \(\sum_\tau r_\tau = 7 r_{\text{base}}\). During target training, the backward pass gradient tensor \(G\) is intercepted prior to projection, partitioned into contiguous column blocks of dimension \(B\), and factored into a scalar \(L_2\) block-magnitude matrix \(M\) and a unit-norm directional matrix \(V\). The high-dimensional directional matrix \(V\) is compressed through the module-specific low-rank projection matrix \(P_\tau\), while the compact magnitude matrix \(M\) is tracked independently in full rank, after which both branches are updated by their respective Adam optimizers and recomposed into parameter updates via broadcast scaling.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Backward Gradient G ∈ R^{m×n}"] --> B["Block-wise Magnitude-Direction Decomposition<br/>Partition rows into 1×B blocks separating M and V"]
B --> C["Static Sensitivity-Driven Module-Aware Rank Allocation<br/>Reallocate rank towards bottleneck receiver modules based on S_τ"]
C --> D["Dual-Branch Optimizer Update & Gradient Recomposition<br/>Low-rank direction projection + full-rank scalar magnitude tracking"]
D --> E["Model Parameter Update W ← W - η · ΔW"]
Key Designs¶
1. Block-wise Magnitude-Direction Decomposition: Isolating Local Energy from Subspace Attenuation Directly projecting unnormalized gradient matrices forces local coordinate magnitude and high-dimensional direction to compress simultaneously into a shared subspace, severely dampening weaker yet crucial gradient features. Conversely, naive row-wise normalization imposes an artificial coupling across heterogeneous channel features, whereas fine-grained element-wise normalization incurs prohibitive memory and computation overheads. MoARa introduces a balanced block-wise decomposition: for a gradient matrix \(G \in \mathbb{R}^{m \times n}\), if the chosen block size \(B \nmid n\), the column dimension is right-padded with zeros to \(n_{\text{pad}}\), partitioning each row into \(k = n_{\text{pad}} / B\) contiguous sub-vectors \(g_{i,j} = (G_{\text{pad}})_{i, jB:(j+1)B} \in \mathbb{R}^B\). For each block, scalar \(L_2\) magnitude and unit direction are extracted as: $\(m_{i,j} = \|g_{i,j}\|_2, \quad (V_{\text{pad}})_{i, jB:(j+1)B} = \frac{g_{i,j}}{m_{i,j} + \epsilon}\)$ where \(\epsilon = 10^{-12}\). Gathering all scalars yields the compact magnitude state \(M \in \mathbb{R}^{m \times k}\), satisfying \(G_{\text{pad}} \approx \text{Rep}_B(M) \odot V_{\text{pad}}\). Because the scalar magnitude term cancels algebraically when evaluating the block-wise cosine alignment \(\cos(G_{i,b}, \tilde{G}_{\text{recon}, i,b}) = \cos(v_{i,b}, \tilde{v}_{i,b})\), projection-induced geometric distortion is isolated entirely to the directional branch, leaving local scale information uncorrupted.
2. Static Sensitivity-Driven Module-Aware Rank Allocation: Relieving Structural Bottlenecks without Momentum Disruption
Transformer modules fulfill sharply divergent computational roles: attn.q and attn.k compute routing affinity scores whose exact representational capacity can tolerate rank compression, whereas mlp.down serves as the primary information bottleneck compressing expanded intermediate features back to the hidden dimension. To measure this projection distortion objectively, MoARa evaluates the expected cosine similarity between full gradients and reconstructed gradients:
$\(S(W, t) = \cos\left(G_t(W), \tilde{G}_{\text{recon}, t}(W)\right), \quad S_\tau = \mathbb{E}_{t, W \in \tau}[S(W, t)]\)$
Empirical profiling consistently places mlp.down at the lowest alignment rank (\(S_\tau \approx 0.69 \sim 0.73\)), while single-module rank-reduction ablations reveal that halving the rank of attn.q or attn.k produces negligible degradation in final perplexity. MoARa designates the donor set \(\mathcal{D} = \{\text{attn.q}, \text{attn.k}\}\) and receiver set \(\mathcal{R} = \{\text{mlp.down}\}\), setting transfer magnitude \(\Delta r = r_{\text{base}} / 2\). Ranks are adjusted via \(r_\tau \leftarrow r_{\text{base}} - \Delta r\) for \(\tau \in \mathcal{D}\) and \(r_\tau \leftarrow r_{\text{base}} + \Delta r \cdot |\mathcal{D}| / |\mathcal{R}| = 2 r_{\text{base}}\) for \(\tau \in \mathcal{R}\), mathematically preserving the exact total budget \(\sum_\tau r_\tau = 7 r_{\text{base}}\). Fixing this allocation statically prior to pre-training avoids mid-training optimizer dimension changes, which would otherwise reset Adam's accumulated first- and second-moment statistics and derail convergence.
3. Head-Dimension-Informed Block Sizing: Balancing Statistical Noise against Subspace Coupling The choice of block granularity \(B\) governs a fundamental statistical trade-off in gradient space. When \(B\) is excessively small (reaching the limit \(B=1\)), the magnitude collapses to absolute values while the direction reduces to a sign matrix (\(\pm 1\)), destroying coordinate-wise scale variation and rendering small-sample \(L_2\) norm variance unacceptably high (\(\text{Std}[m_{i,b}]/\mathbb{E}[m_{i,b}] = \mathcal{O}(B^{-1/2})\)). Conversely, when \(B\) is overly large (approaching full-row normalization \(B=n\)), outlier coordinates across concatenated heads dominate the block norm, artificially attenuating subdominant channels through column-axis subspace coupling. MoARa anchors \(B\) to the neighborhood of the attention head dimension \(d_{\text{head}}\), specifically within the robust operating plateau \(B \in [d_{\text{head}}/2, 2d_{\text{head}}]\). This architecture-informed scale suppresses finite-sample estimation variance while maintaining clean structural boundaries between distinct attention subspaces.
Loss & Training¶
The framework trains autoregressively under standard cross-entropy loss. AdamW is configured with \(\beta_1 = 0.9, \beta_2 = 0.999\), zero weight decay, and \(\epsilon = 10^{-6}\). Peak learning rate is set to 0.01 (0.005 for 7B scale), with 10% linear warmup and cosine decay to a 0.1 minimum ratio. Truncated SVD update frequency is set to \(T_{\text{SVD}} = 200\). The magnitude optimizer state is tracked in FP32, while recomposed gradient updates are cast to BF16 prior to weight application. All models run under standard torch.compile graph optimization.
Key Experimental Results¶
Main Results¶
Evaluated on the English C4 pre-training benchmark across Llama 2 (350M, 1B, 7B) and contemporary open architectures (Llama 3.2 300M, Qwen2.5 350M, Qwen3 350M, and DeepSeek-V2 350M dense variant). On Llama 2 7B, MoARa delivers substantial convergence speedups, and on 1B checkpoints after 100k steps, downstream zero-shot reasoning benchmarks reflect consistent parity or improvement.
| Model / Benchmark | Metric | GaLore (Base) | MoARa (Ours) | Relative Gain / Speedup |
|---|---|---|---|---|
| Llama 2 7B (Convergence Steps) | Steps-to-target ↓ | 150k | 94k | -37.4% (1.60× step speedup) |
| Llama 2 7B (Wall-Clock Schedule) | Wall-clock time ↓ | 168.3h | 110.9h | -34.1% (57.4 hours saved) |
| Llama 2 7B (Iso-Step Perplexity) | 150k Valid PPL ↓ | 13.38 | 13.01 | -2.8% perplexity reduction |
| Llama 2 1B (Language Modeling) | C4 PPL (wppl) ↓ | 86.88 ± 5.61 | 82.62 ± 2.84 | -4.90% (3/3 seed wins) |
| Llama 2 1B (Language Modeling) | WikiText2 PPL (wppl) ↓ | 64.71 ± 4.44 | 60.27 ± 1.45 | -6.86% (2/3 seed wins) |
| Llama 2 1B (Reasoning) | HellaSwag (acc_n) ↑ | 0.386 ± 0.002 | 0.398 ± 0.004 | +3.11% (3/3 seed wins) |
| Llama 2 1B (Commonsense) | WinoGrande (acc) ↑ | 0.513 ± 0.012 | 0.531 ± 0.007 | +3.51% (3/3 seed wins) |
| Llama 2 1B (Question Answering) | BoolQ (acc) ↑ | 0.546 ± 0.046 | 0.558 ± 0.007 | +2.20% (2/3 seed wins) |
Ablation Study¶
Cross-method ablation conducted on Llama 2 350M (60k steps) across six low-rank pre-training hosts, isolating the individual and combined effects of rank reallocation (+Rank), block-wise decomposition (+Decom), and the integrated MoARa framework (+Full):
| Host Optimizer | Configuration | Final Valid PPL ↓ | Steps-to-Target ↓ | Wall-Clock to Target ↓ |
|---|---|---|---|---|
| GaLore | Vanilla Host | 19.48 | 60k (—) | 7.13h (—) |
| + Rank (Allocation Only) | 19.27 | 47k (-22.0%) | 5.58h (-21.7%) | |
| + Decom (Decomp Only) | 19.45 | 54k (-9.6%) | 7.06h (-1.0%) | |
| + Full (MoARa Integrated) | 19.04 | 41k (-31.7%) | 5.29h (-25.8%) | |
| Fira | Vanilla Host | 16.85 | 60k (—) | 7.10h (—) |
| + Rank | 16.79 | 55k (-8.3%) | 6.60h (-7.0%) | |
| + Decom | 16.75 | 53k (-11.7%) | 6.79h (-4.4%) | |
| + Full (MoARa Integrated) | 16.70 | 51k (-15.0%) | 6.65h (-6.3%) | |
| OSD | Vanilla Host | 21.66 | 60k (—) | 8.78h (—) |
| + Rank | 21.27 | 44k (-26.7%) | 6.97h (-20.6%) | |
| + Decom | 20.76 | 37k (-38.3%) | 5.41h (-38.4%) | |
| + Full (MoARa Integrated) | 20.42 | 35k (-41.7%) | 5.52h (-37.1%) | |
| Q-GaLore | Vanilla Host | 20.64 | 60k (—) | 8.27h (—) |
| + Rank | 20.45 | 45k (-25.0%) | 6.19h (-25.2%) | |
| + Decom | 20.81 | Target Not Reached | Target Not Reached | |
| + Full (MoARa Integrated) | 20.46 | 45k (-25.0%) | 6.26h (-24.3%) | |
| SubTrack++ | Vanilla Host | 16.19 | 60k (—) | 6.28h (—) |
| + Rank | 15.57 | 43k (-28.3%) | 4.50h (-28.4%) | |
| + Decom / + Full | 28.29 / 19.17 | Incompatible / Diverged | Incompatible / Diverged |
Key Findings¶
- Module-aware rank allocation exhibits invariant positive transfer: Across all six independent low-rank optimizer hosts, introducing rank reallocation alone consistently reduced optimization steps by 8.3% to 28.3% and wall-clock schedules by 7.0% to 28.4%, validating that module sensitivity asymmetry is an intrinsic architectural property of Transformer decoders rather than an artifact of GaLore.
- Decomposition efficacy depends strictly on host projection dynamics: Block-wise decomposition yields strong compounding gains on methods with static or discretely recomputed projection bases (GaLore, Fira, OSD), reaching up to a 41.7% step reduction on OSD. However, it fails to stabilize on continuous Grassmannian tracking methods (SubTrack++, LDAdam) or INT4-quantized projection schemes (Q-GaLore).
- Negligible peak memory overhead under graph compilation: In compiled execution (
torch.compile), intermediate tensor buffers are largely fused, restricting the 7B peak allocated memory overhead to +0.04 MiB (+0.00%) and peak reserved memory increase to +0.20%, while persistent optimizer states grow by only 1.25 GB (+17.4%).
Highlights & Insights¶
- Gradient-Side Pre-Projection Decoupling: While prior normalization techniques (e.g., Weight Normalization, DoRA) operate on static weight parameters, MoARa demonstrates that decoupling gradients before projection prevents local magnitude variations from being submerged by dominant directional eigenvalues.
- Warmup-Sufficient Static Profiling: Profiling for as few as 1k steps (requiring only ~11 minutes at 350M scale) yields the exact donor-receiver ranking as an exhaustive 60k-step run, allowing practitioners to establish stable rank schedules with zero runtime hyperparameter search overhead.
- Architecture-Informed Discretization Plateau: Establishing block sizes within \([d_{\text{head}}/2, 2d_{\text{head}}]\) naturally bridges the statistical trade-off between small-sample noise and multi-head subspace cross-talk, offering a robust plug-and-play heuristic across diverse model families.
Limitations & Future Work¶
- Lack of a Continuous Dynamic Rank Rescheduling Mechanism: The static allocation avoids momentum re-initialization instability, but cannot adapt if a module's optimization saturation shifts dynamically across later pre-training phases.
- Host-Compatibility Boundary in Continuous Subspace Tracking: The scalar magnitude state cannot easily co-transport with continuously rotating projection bases on Grassmannian manifolds (SubTrack++), demanding future theoretical work on geodesic magnitude transport.
- Absence of Non-Convex Convergence Rate Bounds: The theoretical foundation relies on cosine alignment and information geometry; bounding the exact convergence rate of dual-branch Adam updates under low-rank projection remains an open theoretical challenge.
Related Work & Insights¶
- vs GaLore (Zhao et al., 2024): GaLore enforces a rigid uniform projection rank across all layers and compresses unnormalized gradients; MoARa addresses GaLore's slow convergence by reallocating capacity from routing modules to bottleneck modules and preserving per-block energy scales.
- vs Fira (Chen et al., 2025): Fira reconstructs full-rank updates using matrix-level norm scaling; MoARa decouples magnitude and direction locally before projection at the head-block granularity, showing complementary benefits when paired with Fira.
- vs AdaLoRA / DoRA (Zhang et al., 2023; Liu et al., 2024a): AdaLoRA conducts dynamic SVD rank reallocation for parameter-efficient fine-tuning, and DoRA decomposes fine-tuning weight updates; MoARa pioneers the translation and stabilization of these principles to from-scratch low-rank gradient pre-training and Adam optimizer momentum dynamics.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Pioneers gradient-side magnitude-direction decomposition and architecture-informed rank reallocation in low-rank pre-training]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 300M–7B scales, 5 model architectures, and 6 diverse low-rank pre-training baselines]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity, rigorous mathematical formulations, and thorough empirical-theoretical grounding]
- Value: ⭐⭐⭐⭐⭐ [Directly saves 34%–37% GPU hours in foundational model training while preserving extreme optimizer memory efficiency]