BeamLoRA: Beam-Constraint Low-Rank Adaptation

Conference: ACL 2025
arXiv: 2502.13604
Code: https://github.com/gccnlp/BeamLoRA
Area: Model Compression
Keywords: LoRA, parameter-efficient fine-tuning, rank importance, beam search, dynamic allocation

TL;DR

BeamLoRA observes that the importance of different ranks in LoRA modules varies significantly and evolves dynamically during training. Inspired by beam search, it proposes to dynamically evaluate rank importance inside the training process, prune unimportant ranks, and expand the parameter space for important ranks. This improves performance under a fixed total rank, consistently outperforming LoRA and its variants across 12 datasets on three base models.

Background & Motivation

Background: LoRA is the most popular PEFT method, approximating weight updates through low-rank matrices \(BA\). Existing improvements include AdaLoRA (allocating ranks across modules) and DoRA (decoupling magnitude and direction).

Limitations of Prior Work: Existing methods treat each rank as a homogeneous unit and allocate the same parameter space to all ranks. However, experiments reveal that the importance of different ranks (measured by the Frobenius norm of \(\Delta w_i = b_i a_i\)) varies drastically—pruning low-importance ranks barely affects performance, while a few high-importance ranks determine most of the overall performance.

Key Challenge: The optimization space for important ranks is restricted to a single row and column, while unimportant ranks waste the same parameter budget.

Goal: To reallocate parameter space from unimportant ranks to important ranks under a fixed total rank budget.

Key Insight: Treating LoRA modules as beams in beam search—each rank represents a sub-solution, and the training process is a search for the optimal combination of sub-solutions.

Core Idea: Utilizing a learnable score vector to evaluate rank importance, periodically pruning low-score ranks, and duplicating the parameters of high-score ranks to expand their space.

Method

Overall Architecture

BeamLoRA inserts a learnable score vector \(\mathbf{s} \in \mathbb{R}^r\) in the middle of standard LoRA, modifying the forward pass to \(y = W_0 x + B(\mathbf{s} \odot A)x\). Every \(\Delta t\) steps, scores are evaluated to prune the top-K lowest-scoring ranks (setting them to zero) and duplicate the parameters of the top-K highest-scoring ranks to the pruned positions (expanding the parameter space of important ranks).

Key Designs

1. Learnable Importance Evaluation:

   • Function: Inserts a score vector between \(B\) and \(A\), which scales the output of each rank after softmax normalization.
   • Mechanism: \(s_i\) is trained jointly with LoRA parameters via gradient descent; if a rank is important, \(s_i\) is amplified.
   • Design Motivation: To avoid the high overhead of offline Frobenius norm computation (hundreds of LoRA modules \(\times r\) \(d \times k\) matrices), as the score vector introduces almost zero extra parameters and computational cost.

2. Pruning and Expansion:

   • Function: Periodically reallocates parameter space from low-importance ranks to high-importance ranks.
   • Mechanism: Selects Top-K low-scoring ranks to set to zero (pruned) and duplicates the parameters and optimizer states of the Top-K high-scoring ranks to the pruned positions (expanded). Historical parameters (instead of current ones) are used to break symmetry, enabling the expanded ranks to optimize independently.
   • Design Motivation: Equivalent to providing multiple independent "search paths" for important ranks to optimize, increasing the search width within the solution space.

3. Dynamic Top-P Thresholding:

   • Function: Adaptively determines the value of \(K\) for each pruning/expansion step.
   • Mechanism: Analogous to Top-P (nucleus sampling) in text generation, rankings are sorted by score and truncated when the cumulative score reaches a threshold \(p\). \(p\) transitions from loose to tight (conservative initially, aggressive later) and adapts to different modules.
   • Design Motivation: Avoids a fixed \(K\) value—different modules and training stages require varying degrees of pruning intensity.

Key Experimental Results

Main Results

LLaMA2-7B Mathematical Reasoning (trained on MetaMathQA):

Method	GSM8K	MATH	Trainable Param Ratio
Full FT	68.5	17.4	100%
LoRA (r=16)	66.5	15.3	2.4%
DoRA	67.0	15.8	2.5%
AdaLoRA	66.8	15.5	2.4%
BeamLoRA	68.2	16.9	2.4%

Approaching full fine-tuning performance using only 2.4% parameters (+1.57% avg accuracy over LoRA).

Ablation Study

Configuration	GSM8K	Description
BeamLoRA (full)	68.2	Full method
w/o Pruning & Expansion (Score only)	67.2	Score itself is helpful but inferior to pruning and expansion
w/o Historical params to break symmetry	67.5	Symmetry breaking is crucial
Fixed K (non-dynamic Top-P)	67.6	Dynamic threshold is superior

Key Findings

• Rank importance differences are not distinct early on, stabilizing after about 30-40% of the training process—hence, pruning and expansion require a delayed start.
• Expanded ranks discover new solution paths through independent optimization—it is not simple copying, but a genuine increase in the search space.
• Consistently outperforms LoRA/DoRA/AdaLoRA across three base models (LLaMA2-7B, Mistral-7B, Qwen1.5-7B) and 12 datasets.

Highlights & Insights

• Beam Search Analogy: Analogizing the optimization process of LoRA to beam search, where rank = sub-solution, and pruning/expansion = beam width adjustment. This analogy is intuitive and guides the concrete technical design.
• Almost Zero Extra Cost: The score vector contains only \(r\) scalar parameters, making computational and storage overhead negligible.
• Broad Compatibility: The method is orthogonal to approaches like DoRA and can be used in combination.

Limitations & Future Work

• No Combination with Cross-Module Rank Allocation: BeamLoRA reallocates inside modules, while AdaLoRA reallocates across modules. They are orthogonal and can be combined.
• Time-Window Selection for Historical Parameters: The depth of history from which parameters are retrieved to break symmetry has not been studied in-depth.
• Validated Only on 7B Models: Performance on larger models remains unexplored.

Related Work & Insights

• vs LoRA: BeamLoRA improves performance under a fixed rank budget by reallocating the parameter space.
• vs AdaLoRA: AdaLoRA allocates the number of ranks across modules, while BeamLoRA allocates rank space within modules—they are orthogonal.
• vs DoRA: DoRA decouples magnitude and direction, while BeamLoRA decouples rank importance—they can also be combined.

Rating

• Novelty: ⭐⭐⭐⭐ Novel beam search analogy; insights into rank importance analysis are valuable.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 3 models, 12 datasets, multi-dimensional ablation.
• Writing Quality: ⭐⭐⭐⭐ Clear analysis and intuitive visualizations.
• Value: ⭐⭐⭐⭐ Provides a plug-and-play improvement to LoRA with almost zero extra cost.

Related Papers

• [ACL 2025] CoLA: Collaborative Low-Rank Adaptation
• [ACL 2025] Low-Rank Interconnected Adaptation across Layers
• [ACL 2025] TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition
• [NeurIPS 2025] GoRA: Gradient-Driven Adaptive Low Rank Adaptation
• [ACL 2025] MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning