Skip to content

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

Conference: ICML2025
arXiv: 2506.06280
Code: GitHub - FARMS
Area: Model Compression
Keywords: ESD, Heavy-Tailed Self-Regularization, Random Matrix Theory, Aspect Ratio Bias, Layer-wise Hyperparameter Allocation

TL;DR

The paper proposes FARMS (Fixed-Aspect-Ratio Matrix Subsampling) to eliminate the aspect ratio bias in weight eigenspectrum analysis via fixed-aspect-ratio submatrix sampling, thereby significantly improving HT-SR-based layer-wise learning rate allocation and model pruning.

Background & Motivation

Background: Why Analyze the Weight Eigenspectrum

In recent years, many studies have utilized the empirical spectral density (ESD) of weight matrices to diagnose the training quality of neural networks. From the perspective of Heavy-Tailed Self-Regularization (HT-SR): - A "heavier-tailed" ESD often corresponds to more adequately trained layers. - Under-trained layers typically perform worse on metrics like PL_Alpha.

Such analyses have been applied to: - Layer-wise learning rate allocation (e.g., TempBalance) - Layer-wise pruning ratio allocation (e.g., AlphaPruning) - Layer-wise adjustment in SciML training/fine-tuning

Core Problem: Aspect Ratio Bias Long Ignored

Existing methods often assume that the ESDs of different layers are directly comparable, but the paper points out that this does not hold theoretically.

The reasons are: - When matrices originate from random initialization, their spectral shapes are constrained by the Marchenko-Pastur (MP) distribution. - The MP distribution explicitly depends on the aspect ratio \(\gamma=m/n\). - That is, even with identical training quality, different aspect ratios lead to differences in ESD shapes.

As a result: - Certain layers with large aspect ratios (e.g., 512x100) are likely to be misidentified as "under-trained". - Consequently, this misleads layer-wise hyperparameter allocation. - Ultimately, this degrades training or pruning performance.

The authors refer to this phenomenon as aspect ratio bias.

Method

Review of Traditional HT-SR Metrics

For a layer weight matrix \(W\), the eigenvalues of \(W^\top W\) are first computed, followed by fitting the power-law tail. The commonly used Hill estimator is given by:

\[ \text{PL\_Alpha\_Hill}=1+\frac{k}{\sum_{i=1}^{k}\ln\frac{\lambda_{n-i+1}}{\lambda_{n-k}}} \]

Empirically, a larger PL_Alpha is often interpreted as insufficient training of the corresponding layer.

Core Idea of FARMS

Instead of performing spectral analysis directly on the original large matrix, FARMS: 1. Slices each layer's weight matrix into multiple submatrices using a sliding window. 2. Standardizes all submatrices to a fixed aspect ratio \(Q=m'/n'\). 3. Individually computes the ESDs of the submatrices. 4. Averages these ESDs and then estimates the heavy-tailed indicators.

The benefits of this approach are: - Standardizing the "geometric shape differences" beforehand. - Comparing the "training structure differences" subsequently. - Making cross-layer comparisons fairer.

Algorithmic Flow (Intuitive Version)

  1. Input the layer weight \(W_i\in\mathbb{R}^{m\times n}\).
  2. Clip overlapping submatrices \(W_{i1},W_{i2},...,W_{il}\) using a fixed window size.
  3. Ensure each submatrix has the same aspect ratio (e.g., close to 1).
  4. Compute the ESD of each submatrix individually.
  5. Average the ESDs and then calculate metrics such as PL_Alpha_Hill.
  6. Apply this metric to layer-wise learning rate or pruning ratio allocation.

Relationship with Existing Methods

FARMS is an enhancement at the "analysis layer" level, independent of specific backbone networks or restricted to single tasks. Consequently, it can be integrated as a plug-in into methods like TempBalance and AlphaPruning.

Key Experimental Results

LLM Pruning Results (Key Quantitative Gains of the Paper)

Model & Settings PPL of Original Method PPL with FARMS Relative Improvement
LLaMA-7B + SparseGPT, sparsity=0.8 96.02 79.42 17.3%
LLaMA-13B + Magnitude, sparsity=0.7 2029.20 413.76 Significant Decline

These results indicate that, in compression scenarios, the aspect ratio bias directly affects the quality of layer-wise pruning decisions.

Overall Conclusions Across Scenarios

Application Scenario Baseline Changes Brought by FARMS
CV Training (ResNet/VGG, etc.) TempBalance Series More stable layer-wise learning rate allocation, general improvement in classification performance
LLM Pruning AlphaPruning / SparseGPT Combination Further decrease in perplexity, particularly pronounced under high sparsity rates
SciML Fine-tuning TB_Sigmoid Up to roughly 5.66% reduction in error

Key Observations

  1. FARMS is effective across different tasks and models, indicating that the bias issue is a general phenomenon.
  2. Models with many "irregularly shaped layers" often benefit the most.
  3. The method is lightweight, yet exerts a significant impact on downstream optimization decisions.

Highlights & Insights

  1. Formulating a "seemingly statistical detail" as a "source of training decision bias" provides strong insight.

  2. The method is simple to implement and highly compatible. It does not change network structures or training objectives, modifying only the spectral analysis process.

  3. The results hold engineering value. The actual decrease in perplexity in LLM pruning demonstrates that it is not merely a theoretical improvement on paper.

  4. The paper conveys an important methodology: In cross-layer comparisons, geometric/scale calibration must be performed before deriving structural conclusions.

  5. This work may influence a broader range of spectral analysis applications, beyond HT-SR.

Limitations & Future Work

  1. Additional computational overhead. Repeated spectral calculations on submatrices still incur overhead on extremely large models.

  2. Submatrix strategies can still be optimized. For instance, window size, stride, and overlap ratio affect estimation stability and efficiency.

  3. Currently, the focus is primarily on PL_Alpha-type metrics. The benefits when integrated with other spectral statistics warrant further systematic study.

  4. A gap remains between theory and practice. A more detailed theoretical explanation is needed to clarify why fixed aspect ratios remain robust across different architectures.

  5. For extremely small layers or highly sparse layers, subsampling statistical stability requires more boundary experiments.

  • In line with the HT-SR/WeightWatcher series, belonging to the direction of "more reliable spectral diagnostic tools".
  • Its relationship with TempBalance and AlphaPruning is enhancement rather than replacement: It improves the "diagnostic input quality," correcting layer-wise strategies from the source.
  • Insights for subsequent research:
  • Attempting to integrate FARMS into automated optimizers (layer-wise LR, layer-wise WD, layer-wise sparsity).
  • Extending it to more irregularly shaped architectures, such as MoE and multi-branch networks.
  • Exploring a joint framework of "unbiased spectral diagnostics + neural architecture search".

Rating

  • Novelty: ⭐⭐⭐⭐☆ (4.0/5)
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ (4.5/5)
  • Writing Quality: ⭐⭐⭐⭐☆ (4.0/5)
  • Value: ⭐⭐⭐⭐⭐ (5.0/5)

Overall Evaluation: This is a work with "accurate problem definition + clear engineering implementation". Instead of gaining performance breakthroughs via complex new models, it significantly improves existing methods by correcting statistical bias, making it a highly valuable methodology paper for reuse.