Gradient Starvation in Low-Rank Language Model Pretraining¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ovasoft/rank_dynamics
Area: Model Compression
Keywords: Low-Rank Pretraining, Gradient Starvation, Representation Alignment, Effective Rank, Spectral Geometry
TL;DR¶
This paper reveals that the representational and performance deficit in low-rank language model pretraining is not inherited from initialisation, but is sustained by an escalating deficit in gradient effective rank (gradient starvation), while demonstrating that optimisation gains (perplexity) dissociate from representational alignment.
Background & Motivation¶
Efficient pretraining of modern language models increasingly relies on low-dimensional constraints to curb the prohibitive costs of memory and computation. Low-rank factorisations such as \(W = BA\) (\(r \ll \min(m, n)\)), gradient projection methods like GaLore, and structured accumulation schemes like ReLoRA have emerged as viable tools for training under severe hardware limitations. Despite continuous algorithmic modifications, low-rank (LR) pretraining persistently underperforms its full-rank (FR) counterpart across model architectures and training scales. The literature has responded almost entirely from an engineering standpoint—iterating on empirical tricks to narrow the performance gap—while leaving the foundational mechanistic question unaddressed: how does the low-rank constraint reshape internal representations during optimisation, and why does this representational damage resist correction?
The core tension lies in the untested assumption that the deficiency of low-rank models originates from spectral defects at initialisation or merely reflects a trivial deficit in trainable parameter counts. However, the causal distinction between initialisation artifacts and active optimisation constraints has never been empirically decoupled. If full-rank training continuously explores an isotropic, high-dimensional update manifold while low-rank training is confined to a severely restricted update subspace, the representational trajectories of the two regimes must inevitably diverge. Furthermore, when engineering interventions succeed in reducing training perplexity, it remains fundamentally unverified whether the underlying model truly recovers full-rank representational fidelity and downstream functional transfer capability.
To resolve these mechanistic questions, this paper investigates the training dynamics of GPT-2 on the BabyLM Strict-Small benchmark across six targeted objectives. The authors introduce spectrum-matched initialisation to eliminate step-zero disparities, deploy gradient truncation (GradTrunc) and subspace reset (ReLoRA) to isolate gradient dimensionality from weight parameterisation, and conduct cross-domain checks on ViT/CIFAR-10. Core idea: The representational split between low-rank and full-rank language models is not fixed at initialisation, but is dynamically sustained and widened during training by gradient effective rank starvation; moreover, optimisation success (perplexity) dissociates from representational alignment, demonstrating that gradient dimensionality is the primary causal driver of low-rank representational degradation.
Method¶
Overall Architecture¶
The paper designs an empirical and causal intervention framework across six mechanistic objectives to dissect the temporal evolution, spectral geometry, gradient causality, and functional consequences of low-rank pretraining. The diagnostic workflow begins with spectrum-matched initialisation (via truncated SVD) to align the starting point with full-rank weights at step zero. It tracks gradient dimensionality via effective rank (eRank) throughout training and identifies stabilisation regimes. It then isolates gradient dimensionality from weight factorisation using gradient truncation (GradTrunc) on full-rank weights versus periodic subspace expansion (ReLoRA) under fixed low-rank weights. Finally, it probes the dissociation between optimisation metrics and geometric alignment using dynamic spectral normalisation (DSN), validating functional transfer through BLiMP minimal grammatical pairs and mutual k-NN alignment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Pretraining Diagnostic Setup<br/>(GPT-2 on BabyLM / ViT on CIFAR-10)"] --> B["Gradient Effective Rank Starvation Diagnosis<br/>Track eRank dynamics and exhaustion stabilisation"]
B --> C["Gradient Dimensionality Causal Decoupling<br/>GradTrunc gradient truncation vs. ReLoRA subspace refresh"]
C --> D["Spectral Correction & Optimisation-Alignment Dissociation<br/>t* singular value flattening vs. dynamic spectral normalisation DSN"]
D --> E["Layer Reinitialisation & Functional Probing<br/>k-sweep block reset & BLiMP grammatical evaluation"]
E --> F["Output: Gradient dimensionality governs representational split<br/>Perplexity improvements dissociate from representation alignment"]
Key Designs¶
1. Gradient Effective Rank Starvation Diagnosis: Uncovering the Sustained Driver of Representational Divergence
To quantify the dimensionality of the optimisation search space available to the optimiser at each update step, the authors measure the effective rank (eRank) of the gradient matrix \(G\), defined as the exponential Shannon entropy of its normalised singular value spectrum: $\(eRank(G) = \exp\left(-\sum_{i} p_i \ln p_i\right), \quad p_i = \frac{\sigma_i(G)}{\sum_{j} \sigma_j(G)}\)$ In full-rank language pretraining, the gradient effective rank dynamically expands from 306 to 510 over the course of training, indicating that gradient energy spreads across progressively richer update directions. In contrast, the low-rank model (\(r=8\)) remains tightly bounded near its structural rank ceiling, causing the gradient richness ratio \(eRank(G_{\text{FR}})/eRank(G_{\text{LR}})\) to escalate from 14.3 to 42.3. To rule out initialisation as the root cause, a spectrum-matched initialisation is constructed by projecting FR's own initial weights onto rank-8 factors via truncated SVD. Although this completely eliminates the FR–LR CKA gap at step zero (reaching 123.6% of the FR–FR baseline), CKA collapses by 78% within just one batch (8,192 tokens) and drops below default initialisation by the end of training. This proves that initialisation offers no lasting protection and that continuous gradient starvation sustains the split.
2. Gradient Dimensionality Causal Decoupling: Separating Update Subspace from Weight Factorisation
To demonstrate that gradient dimensionality is a causally separable driver rather than a trivial corollary of the weight factorisation \(W = BA\), the authors implement two complementary interventions. The forward intervention, GradTrunc, maintains an unfactorised full-rank weight matrix while projecting its gradients onto a fixed rank-8 random orthonormal subspace at every step. The reverse intervention, ReLoRA, enforces a strict rank-8 weight factorisation while merging the adapter into the base weights and reinitialising the update subspace every 5M tokens. GradTrunc achieves a final gradient effective rank of 77.6 and replicates the representational split, closing 50% of the CKA gap to FR without ever factorising weights. Conversely, ReLoRA reaches an effective rank of 60.1 and recovers 70% of the CKA gap while maintaining rank-8 weights throughout. Because neither outcome can be explained by weight factorisation alone, gradient dimensionality is established as the active causal mechanism.
3. Spectral Correction & Optimisation-Alignment Dissociation: Perplexity Gains Without Representation Recovery
To test whether intervening on spectral geometry can reverse representational divergence, the authors apply singular value alpha-flattening at the first checkpoint \(t^*\), enforcing a uniform, high-entropy singular value distribution. In language models, this instantaneous intervention produces a +288% increase in cross-run CKA and a -62% reduction in validation perplexity, demonstrating that early spectral geometry is causally operative. However, standard gradient flow overwrites this correction within subsequent steps. To investigate whether sustained intervention maintains alignment, the authors introduce Dynamic Spectral Normalisation (DSN), applying static uniform, dynamic FR-matched, or flat singular value scaling every 1,000 steps. Strikingly, while DSN variants lower perplexity by 75–77% (slashing PPL from 456.6 to 109.7–117.2), final CKA values remain clustered around the untreated LR baseline (0.031–0.040). This reveals a profound dissociation: substantial optimisation success can be achieved without bringing internal representations any closer to the full-rank manifold.
4. Layer Reinitialisation & Functional Probing: Downstream Consequences of Representational Split
To assess whether the damage inflicted by low-rank pretraining is post-hoc recoverable and carries functional consequences, the authors perform a \(k\)-sweep reinitialisation of the first \(k\) transformer blocks followed by fine-tuning, alongside probing across 1,600 BLiMP minimal grammatical sentence pairs. In language models, recoverability scales monotonically with the number of reinitialised layers (\(k=2\) closes 31.2%, \(k=4\) closes 54.6%, and reinitialising all 11 blocks closes 73.0%), yet a substantial 58% performance deficit remains unrecovered under convergence-budget full fine-tuning, demonstrating that the low-rank model resides in an isolated basin of attraction. Downstream functional probing shows that CKA alignment correlates moderately with BLiMP grammatical accuracy (\(r = 0.40\)), whereas validation perplexity shows negligible correlation (\(r = -0.08\)). Crucially, DSN variants that achieve superior perplexity perform at chance on BLiMP (51.4%, binomial test \(p > 0.05\)), while the untreated LR baseline with far worse perplexity maintains statistically significant grammatical competence (55.6%, \(p < 0.05\)), an observation corroborated by independent mutual k-NN alignment.
Loss & Training¶
The primary architecture is a standard GPT-2 configuration (12 layers, 768 hidden dimension, 12 attention heads, ~85M parameters) pretrained on the BabyLM Strict-Small corpus (50M tokens) for 5 epochs. Optimization is conducted using AdamW with peak learning rate \(10^{-4}\), 500-step linear warm-up, cosine learning rate decay, batch size 64, sequence length 128, and BF16 mixed precision on NVIDIA A100 GPUs. In all LR configurations, linear projections in multi-head attention and feed-forward networks are replaced by factorised matrices \(W = BA\) across \(r \in \{8, 16, 32, 64, 128, 256\}\). Identical hyperparameters are maintained across FR and LR models (except for the LR-tuned baseline, which sweeps learning rate and initialisation scale), ensuring that observed differences are strictly attributable to rank and gradient constraints.
Key Experimental Results¶
Main Results¶
The primary benchmark evaluations compare the full-rank baseline against standard low-rank setups, gradient causal interventions, and spectral normalisation variants. Metrics encompass Centered Kernel Alignment (CKA to FR), BLiMP zero-shot accuracy, validation perplexity (Val. PPL), gradient effective rank growth ratio (final vs. initial), and consecutive-checkpoint stability (Self-CKA).
| Model Config | CKA to FR | BLiMP Acc | Val. PPL | eRank Growth | Self-CKA | Note / Key Behavior |
|---|---|---|---|---|---|---|
| FR (Full-Rank Baseline) | 0.9800 | 0.6338 | 46.98 | 5.98× | 1.0000 | Gradient eRank expands to 510; performance and alignment ceiling |
| LR_r8 (Baseline LR) | 0.0349 | 0.5563 | 456.60 | 0.65× | 0.7050 | Experiences severe gradient starvation; rapid divergence, high PPL |
| GradTrunc8 | 0.0645 | 0.5775 | 72.81 | 2.48× | 0.8499 | Full-rank weights + rank-8 projected gradient reproduces split (closes 50% CKA) |
| ReLoRA_r8 | 0.0527 | 0.4950 | 604.58 | 1.91× | 0.8839 | Periodic subspace reset restores gradient dimensionality (closes 70% CKA) |
| LR_specmatch8 | 0.0287 | 0.5325 | 438.75 | 4.73× | 0.6758 | Initialises at 123.6% CKA; collapses within 1 batch, higher final divergence |
| LR_r8_tuned | 0.0398 | 0.5200 | 138.24 | 0.62× | 0.7300 | Hyperparameter tuning cuts PPL by 69.7% while closing only ~8% CKA gap |
| LR_r8_dsn_static | 0.0313 | 0.5150 | 117.18 | 0.93× | 0.9507 | Static spectral normalisation; low PPL (117) but BLiMP reduces to chance |
| LR_r8_dsn_dynamic | 0.0400 | 0.5144 | 114.73 | 0.94× | 0.9556 | Dynamically matched to FR singular values; PPL improves, CKA unchanged |
| LR_r8_dsn_flat | 0.0328 | 0.4875 | 109.67 | 0.83× | 0.9496 | Enforces uniform singular values; lowest PPL (109.67) but fails BLiMP probing |
Ablation Study¶
The impact of distinct spectral correction interventions applied at checkpoint \(t^*\) across both language (GPT-2 on BabyLM) and vision (ViT on CIFAR-10) modalities:
| Domain | Variant | Mean CKA | CKA Gain | Task Metric | Task Gain | Observation & Insight |
|---|---|---|---|---|---|---|
| Language (BabyLM) | Baseline (untreated LR) | 0.082 | — | 489.7 PPL | — | Shows severe early representational separation |
| Language (BabyLM) | Drift-scaled | 0.104 | +27% | 478.5 PPL | -2% | Scaling by measured drift yields marginal improvement |
| Language (BabyLM) | Alpha-flat | 0.318 | +288% | 188.2 PPL | -62% | Restoring isotropic spectrum yields substantial immediate recovery |
| Language (BabyLM) | Combined | 0.198 | +141% | 268.2 PPL | -45% | Underperforms alpha-flat alone; drift scaling adds noise |
| Vision (CIFAR-10) | Baseline (untreated LR) | 0.318 | — | 0.223 Acc | — | Vision baseline at \(t^*\) |
| Vision (CIFAR-10) | Drift-scaled | 0.362 | +14% | 0.234 Acc | +5% | Modest gain in top-1 validation accuracy |
| Vision (CIFAR-10) | Alpha-flat | 0.516 | +62% | 0.253 Acc | +13.5pp | Major boost in representation alignment and accuracy |
| Vision (CIFAR-10) | Combined | 0.470 | +48% | 0.249 Acc | +12pp | Underperforms pure alpha-flattening |
Key Findings¶
- Domain-Specific Gradient Starvation Dynamics: In language modeling, the full-rank gradient effective rank expands over training (306 to 510), causing the gradient richness ratio \(eRank(G_{\text{FR}})/eRank(G_{\text{LR}})\) to widen from 14.3 to 42.3 as an escalating constraint. In vision (ViT on CIFAR-10), this ratio remains constant across all epochs (CV 2–7%), acting as a fixed constraint.
- Universal Exhaustion Stabilisation in Language: All language training runs (including the full-rank model and all low-rank variants) stabilise via the exhaustion mechanism (\(\text{init-CKA}(t_s) < 0.65\), ranging between 0.44 and 0.67), drifting far from initialisation before step-to-step velocity drops. Unlike vision, which exhibits a layer hierarchy where shallow layers cleanly converge, language models exhibit uniform exhaustion across all 12 layers.
- Fundamental Dissociation Between Optimisation and Geometry: Dynamic spectral normalisation and extensive hyperparameter tuning reduce validation perplexity by up to 77% while leaving CKA representation alignment virtually unchanged. Crucially, CKA correlates positively with downstream BLiMP grammatical accuracy (\(r = 0.40\)), while perplexity exhibits zero correlation (\(r = -0.08\)), confirming that optimizing perplexity alone fails to guarantee representational fidelity.
Highlights & Insights¶
- Causal Decoupling of Gradient Dimensionality: By introducing GradTrunc (unfactorised weights with low-rank gradients) and ReLoRA (factorised weights with refreshed gradient subspaces), the authors prove that gradient dimensionality per se, rather than the factorised weight matrix parameterisation, is the operative causal constraint behind representational divergence.
- Refutation of the Initialisation Myth: Truncated SVD spectrum-matched initialisation demonstrates that even when low-rank and full-rank models start with identical representational geometry at step zero (123.6% CKA), the rank constraint reasserts the split within 8,192 tokens.
- Rethinking Evaluation Benchmarks for Efficient Pretraining: The paper exposes the risk of evaluating low-rank pretraining purely through validation perplexity. Models can exploit distorted representational spaces to minimize token prediction loss while degrading downstream linguistic competence, emphasizing the necessity of evaluating representational trajectories alongside task metrics.
Limitations & Future Work¶
- Scale Constraints: Due to the computational cost of dense checkpoint saving and extensive SVD decomposition, experiments are primarily conducted on GPT-2 (85M) and ViT (4.3M). Whether gradient starvation dynamics behave identically at frontier multi-billion-parameter scales remains an open empirical question.
- Single Optimiser Scope: All core findings are derived using AdamW. Exploring whether alternative optimisers that explicitly manipulate the gradient subspace (e.g., Muon, second-order optimisers, or stochastic projection schemes) can mitigate gradient starvation warrants further study.
- Metric Discrepancies: While mutual k-NN alignment independently confirms the persistent representational gap, it does not perfectly match CKA on the rank ordering of specific interventions (e.g., DSN-static). Incorporating Neural Tangent Kernel (NTK) analysis or broader functional probing suites could provide a richer characterisation.
Related Work & Insights¶
- vs ReLoRA (Lialin et al., 2024): ReLoRA showed empirically that periodically merging low-rank updates into full-rank weights narrows the pretraining gap. This paper provides the underlying mechanistic explanation: ReLoRA mitigates gradient starvation by continuously expanding the effective gradient dimensionality across training (eRank reaches 60.1, closing 70% of the CKA gap).
- vs GaLore (Zhao et al., 2024): GaLore utilizes low-rank gradient projection to compress optimizer memory while maintaining full-rank weights. The GradTrunc experiment in this paper directly proves that restricting gradient updates to a static low-dimensional subspace induces severe representational divergence, explaining why dynamic subspace switching is mandatory in GaLore.
- vs Diehl Martinez et al. (2024): While prior work noted that low effective rank correlates with unstable convergence in small language models, this paper uncovers the dynamic widening of the gradient effective rank gap, establishes its causal autonomy from weight factorisation, and reveals the dissociation between perplexity and representation alignment.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Provides the first rigorous mechanistic explanation of low-rank pretraining degradation via gradient effective rank starvation and spectral causality]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Features an exceptionally complete causal intervention suite—spectrum matching, GradTrunc, ReLoRA, DSN, k-sweep, and BLiMP probing]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity in formulating hypotheses, formalising metrics, and navigating the causal implications of the empirical findings]
- Value: ⭐⭐⭐⭐⭐ [Crucial for the foundational understanding of parameter-efficient pretraining; issues a timely warning against evaluating low-rank models solely by perplexity]