Skip to content

LAK-LoRA: Layer-Wise Adaptive Expert Allocation and Principal Component–Based Feature Separation for Medical Multi-Task Learning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Parameter-Efficient Fine-Tuning, Mixture of Experts, Feature Disentanglement, Singular Value Decomposition, Medical Multi-Task Learning

TL;DR

LAK-LoRA addresses conflicting objectives and capacity mismatches in medical multi-task LLM fine-tuning by introducing depth-adaptive expert capacity scaling alongside an SVD-based principal component feature separator and orthogonality constraints, outperforming existing multi-task PEFT methods without inflating parameter budgets.

Background & Motivation

Large language models have demonstrated immense promise across diverse clinical natural language processing domains, ranging from biomedical question answering and electronic health record extraction to diagnostic intention recognition and medical dialogue generation. However, deploying fine-tuned LLMs in practical clinical environments poses severe challenges due to the sheer diversity of medical tasks. Because full-parameter fine-tuning for each distinct clinical scenario is computationally prohibitive, multi-task parameter-efficient fine-tuning (PEFT) has emerged as an indispensable paradigm for real-world medical applications. Unfortunately, divergent clinical tasks frequently impose conflicting optimization objectives: named entity recognition requires fine-grained local representations, intention classification relies on clear decision boundaries, and report generation prioritizes overarching narrative coherence. This mismatch frequently causes severe negative transfer, commonly termed the "see-saw phenomenon."

While recent LoRA-based multi-task architectures attempt to alleviate parameter conflict by deploying parallel adapters or mixture-of-experts (MoE) routing, existing paradigms suffer from two foundational structural deficiencies. Vertically across network depth, conventional MoE-LoRA methods uniformly assign an identical number of experts to every layer. This homogeneous allocation directly contradicts the hierarchical nature of deep language backbones, where shallow layers capture generic transferable patterns while deeper layers progressively specialize into task-dependent semantic representations. Prior efforts to expand deep-layer capacity rely primarily on rigid, hand-crafted heuristics that fail to adapt across different model scales and layer depths. Horizontally within each layer, current methods treat intermediate representations as indivisible, entangled vectors, forcing adapters to simultaneously process shared medical foundations and highly specialized task signals.

The central insight of this work is grounded in the spectral characteristics of domain-specific data: across diverse clinical tasks, core biomedical concepts and physiological knowledge reside within dominant, high-variance representation subspaces, whereas task-specific requirements inhabit lower-energy residual manifolds. Core idea: by leveraging singular value decomposition to decouple hidden activations into an invariant shared subspace and a task-specific residual subspace, and routing them respectively to a generic expert and depth-adaptively scaled specialized experts under orthogonality constraints, the model eliminates cross-task interference while preserving multi-task representation capacity.

Method

Overall Architecture

At each adaptation layer, the LAK-LoRA information pipeline decouples hidden states and directs them through specialized processing streams before computing the final low-rank update. Specifically, the incoming representation is first processed by an offline-initialized feature separator that projects activations into mutually orthogonal shared and task-specific subspaces. The shared component is deterministically assigned to a single generic expert to model universal medical semantics across all tasks. Simultaneously, the residual task-specific component is routed to an ensemble of specialized experts weighted dynamically via a task-driven gate. Crucially, the total number of experts at each layer follows an adaptive Sigmoid growth curve, smoothly expanding capacity in deeper layers while strictly splitting the global LoRA rank to prevent parameter inflation. Finally, an orthogonality constraint is enforced across specialized experts to eliminate representational redundancy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Hidden State x"] --> B["Principal Component Feature Separator<br/>SVD spectral projection decouples shared and specific subspaces"]
    B -->|Shared Component xs| C["Generic Expert Modeling<br/>Single dedicated expert models universal semantics"]
    B -->|Task-specific Residual xt| D["Layer-Wise Adaptive Expert Allocation<br/>Sigmoid-guided expert capacity scaling across depth"]
    D --> E["Specialized Expert Diversity Constraint<br/>Pairwise cosine orthogonality loss reduces redundancy"]
    C --> F["Dual-Path Aggregation Output<br/>Linear fusion: W0*x + hs + ht"]
    E --> F

Key Designs

1. Principal Component-Based Feature Separator: Resolving Cross-Task Interference via Spectral Subspace Decoupling
To address representation entanglement where universal clinical knowledge and conflicting task objectives contaminate the same activation vector, this design leverages the spectral structure of intermediate states. Before fine-tuning, the model executes a single forward pass over an unannotated sample subset to collect hidden activations across target layers, forming an activation matrix \(Z \in \mathbb{R}^{d \times n}\). After channel-wise mean and variance normalization, singular value decomposition yields \(\tilde{Z} \approx \hat{U} \hat{\Lambda} \hat{V}^\top\). Utilizing the information contribution ratio \(p_i = \sigma_i^2 / \sum_j \sigma_j^2\), a cumulative entropy threshold \(\eta\) determines the minimal shared rank \(r_s\): $\(-\sum_{i=1}^{r_s} p_i \log p_i \ge \eta \cdot H_n\)$ The leading left singular vectors \(\hat{U}_s \in \mathbb{R}^{d \times r_s}\) form the orthogonal projection operator \(P_s = \hat{U}_s \hat{U}_s^\top\). For any runtime hidden vector \(x\), the representation decomposes into a shared component \(x_s = P_s x\) and a task-specific component \(x_t = (I - P_s)x\). Statistical derivation confirms that under the dominant shared clinical covariance assumption, \(\mathrm{Cov}(x_s, x_t) = 0\), guaranteeing zero cross-covariance between the two feature streams.

2. Layer-Wise Adaptive Expert Allocation: Matching Hierarchical Model Semantics with Depth-Aware Capacity Scaling
To resolve the structural mismatch caused by static, uniform expert allocation across heterogeneous model depths, this design adaptively scales expert capacity via a continuous Sigmoid progression. For a backbone with \(L\) layers, the depth adaptation factor \(\alpha_l\) at layer \(l\) is governed by: $\(\alpha_l = \mathrm{sigmoid}\left(-\epsilon + \frac{2\epsilon \cdot l}{L}\right)\)$ where \(\epsilon\) modulates the growth steepness. This factor is indexed into a monotonically increasing discrete sequence \(V\), dynamically selecting the total expert count \(N_l\). Crucially, the total LoRA rank \(r\) is uniformly divided such that each expert possesses rank \(r / N_l\), guaranteeing that the total trainable parameter budget per layer strictly equals standard LoRA (\(r \cdot k + d \cdot r\)). Exactly one expert acts as the generic expert \(E_s^{(l)}\), while the remaining \(N_l - 1\) experts form the specialized expert group \(E_{t,i}^{(l)}\).

3. Expert Diversity Orthogonality Constraint: Suppressing Representational Redundancy in Specialized Expert Ensembles
Specialized experts receive the residual vector \(x_t\) and are weighted by a task-driven gate \(g(e_{\mathrm{task}}) = \mathrm{softmax}(W_g e_{\mathrm{task}} / \tau)\), parameterized by task embedding \(e_{\mathrm{task}} \in \mathbb{R}^{d_e}\). To prevent multiple specialized experts from collapsing into identical update subspaces under overlapping task gradient directions, this design introduces a pairwise parameter orthogonality penalty. Flattening each expert's low-rank matrices into vectors \(v_i, v_j\), the layer-wise penalty calculates the mean squared cosine similarity across all unique pairs: $\(\mathcal{L}_{\mathrm{ortho}}^{(l)} = \frac{1}{\binom{N_l-1}{2}} \sum_{i < j} \left( \frac{v_i \cdot v_j}{\|v_i\| \|v_j\|} \right)^2\)$ This explicit regularization enforces directional divergence during backpropagation, ensuring that deep layers with high expert capacity develop complementary, highly specialized clinical representations.

Loss & Training

The overall training objective combines task-specific supervised prediction loss with the multi-layer expert orthogonality penalty: $\(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{task}} + \lambda \cdot \mathcal{L}_{\mathrm{ortho}}\)$ where \(\mathcal{L}_{\mathrm{task}}\) denotes standard cross-entropy loss averaged across batch samples, \(\mathcal{L}_{\mathrm{ortho}}\) averages the orthogonality loss over all layers containing multiple specialized experts, and \(\lambda\) balances regularization strength. Optimization utilizes AdamW with linear learning rate decay and zero weight decay. LoRA adapters target the multi-head attention projections (q_proj, k_proj, v_proj, o_proj) with default rank \(r=16\). SVD projection matrices are computed once prior to training and frozen throughout fine-tuning and inference, avoiding runtime matrix decompositions and preserving strict numerical stability.

Key Experimental Results

Main Results

On the comprehensive Chinese PromptCBLUE benchmark encompassing 16 diverse clinical NLP tasks, LAK-LoRA was evaluated against full-adapter, multi-adapter, and MoE-based PEFT baselines using the Qwen2.5-7B-Instruct backbone, alongside medical foundation models evaluated via in-context learning (ICL). Representative task scores and the 16-task overall macro averages are summarized below:

Method / Backbone CMeEE-V2 CMeIE-V2 CHIP-CDN CHIP-CTC KUAKE-IR KUAKE-QIC 16-Task Avg. Trainable Params (%)
Qwen-Max (ICL) 49.614 29.552 65.986 70.843 62.315 49.758 49.677 -
Baichuan-M1 (ICL) 43.155 18.004 65.240 54.818 58.370 55.848 45.789 -
LoRA (Single) 66.259 37.740 81.656 87.314 85.721 86.759 68.023 0.1323 × 16
LoRA (Full) 65.338 41.720 80.075 84.975 83.783 88.256 66.841 0.1323
MultiLoRA 66.435 39.350 83.508 88.071 85.992 87.724 68.100 0.4048
MOELoRA 69.392 46.699 82.103 89.399 83.268 78.100 68.966 0.1347
MTL-LoRA 68.126 42.552 84.565 86.552 88.365 86.616 69.244 0.2344
LAK-LoRA (Ours) 70.196 48.012 85.421 90.029 89.916 85.917 70.919 0.1346

In out-of-domain generalization experiments on Qwen2.5-3B-Instruct evaluated across five distinct datasets (CMExam, CMB-Exam, CMedCausal, Med-Dialogue, Yidu-N7K): - LoRA (Full) averaged 37.676; MOELoRA achieved 41.499; MTL-LoRA reached 42.426; DoRA recorded 34.482; AdaLoRA lagged at 16.165. - LAK-LoRA reached 43.993, achieving massive gains on structured extraction tasks like CMedCausal (74.168 vs MTL-LoRA 66.823) and clinical normalization Yidu-N7K (71.886 vs MTL-LoRA 62.028), where structural output constraints cause unregularized baselines to fail.

Ablation Study

On the Qwen2.5-7B-Instruct backbone, isolated component removals and architectural variant sweeps were benchmarked across the 16 PromptCBLUE tasks (standard configuration baseline score: 70.020):

Configuration / Variant 16-Task Avg. Score Performance \(\Delta\) Core Implication
LAK-LoRA (Full) 70.020 - Default configuration (\(\tau=0.1, \eta=0.3, \lambda=0.01, \epsilon=4\))
Fixed expert allocation (\(N_l = 2\)) 69.126 -0.894 Insufficient deep-layer capacity restricts task specialization
Fixed expert allocation (\(N_l = 8\)) 69.196 -0.824 Excessive shallow-layer experts dilute shared feature learning
Fixed expert allocation (\(N_l = 16\)) 69.627 -0.393 High uniform capacity still underperforms depth-aware scaling
w/o Feature separation 69.055 -0.965 Severe drop; representation entanglement triggers cross-task interference
w/o Orthogonality constraint 69.884 -0.136 Specialized experts suffer from redundant parameter updates
Linear expert growth curve 69.585 -0.435 Fails to match the sharp semantic transition in mid-to-deep layers
Exponential expert growth curve 69.565 -0.455 Over-compresses shallow capacity, starving generic concept formation
Generic experts increased to 3 69.069 -0.951 Redundant generic capacity overfits shared features

Key Findings

  • Feature separation serves as the primary barrier against task conflict: Removing the SVD-based feature separator caused the single largest degradation (-0.965 overall average drop), with natural language inference (KUAKE-QQR) plummeting by 6.016 points (77.266 to 71.250). As visualized via t-SNE, task representations in the shared subspace cluster tightly together, whereas the task-specific subspace exhibits clearly demarcated, isolated clusters.
  • Depth-adaptive expert scaling outperforms static uniform allocations: Allocating fewer experts in shallow layers and more in deep layers via the Sigmoid curve consistently surpasses both low-capacity (2 experts) and high-capacity (16 experts) uniform setups. Heatmap analysis shows that shallow layers exhibit coarse-grained expert sharing, while layer 28 displays sharp sparsity, with different tasks activating non-overlapping specialized experts.
  • Superior parameter and runtime efficiency: LAK-LoRA requires only 0.1346% trainable parameters—comparable to standard LoRA and far lower than MultiLoRA (0.4048%) or MTL-LoRA (0.2344%). In inference latency benchmarks on MedQA and MedMCQA, LAK-LoRA requires 1.51 seconds per sample, remaining fully competitive with standard LoRA (1.37 seconds).

Highlights & Insights

  • Spectral statistical decoupling without auxiliary parameter bloat: Instead of adding heavy learnable projections, LAK-LoRA leverages the natural covariance structure of medical language activations to decouple features via SVD, backed by rigorous proof establishing zero cross-covariance.
  • Rank-partitioned MoE ensures zero parameter overhead: By dynamically partitioning the LoRA rank \(r\) across \(N_l\) experts, the architecture grants deep layers high routing degrees of freedom while maintaining an identical parameter budget to single-adapter LoRA.
  • Broad cross-domain transferability: The underlying principle—extracting high-variance shared concepts while adaptively scaling capacity for low-variance task specifics—offers immediate value for multi-modal vision-language alignment and multi-lingual cross-lingual transfer.

Limitations & Future Work

  • Author-acknowledged limitations: Due to computational constraints, evaluations were conducted on models up to 7B parameters (Qwen2.5-7B, ChatGLM-6B, Qwen3-4B); testing on 70B+ scale models remains unverified. Furthermore, the framework is currently restricted to text-only medical tasks, leaving multimodal clinical scenarios unexplored.
  • Potential empirical constraints: The SVD projection operator \(P_s\) is statically computed from a pre-training activation sample. In the presence of extreme distribution shifts between initial samples and streaming downstream fine-tuning data, the static projection may exhibit slight spectral drift.
  • Future directions: Investigating online or momentum-based incremental SVD projection updates during multi-stage fine-tuning, and extending the dual-path decoupling framework to multimodal clinical systems encompassing radiological imaging and genomic sequences.
  • vs. MOELoRA (Liu et al., 2023b): MOELoRA applies uniform expert counts across all layers and feeds entangled hidden states into routers, leading to severe task conflicts; LAK-LoRA introduces depth-adaptive expert capacity and explicit feature separation, surpassing MOELoRA by 1.953 points on PromptCBLUE.
  • vs. MTL-LoRA (Yang et al., 2025): MTL-LoRA requires task-specific diagonal and up-projection matrices that scale linearly with the number of tasks; LAK-LoRA achieves higher performance with 43% fewer trainable parameters via rank-partitioned MoE and spectral projection.
  • vs. HydraLoRA (Tian et al., 2024): HydraLoRA adopts an asymmetric shared-A and routed-B design; LAK-LoRA demonstrates that explicitly separating the feature representation into shared and specific subspaces before routing provides substantially stronger protection against negative transfer.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Combines SVD spectral feature decoupling with layer-wise depth-adaptive MoE scaling in an elegant, mathematically grounded design)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluates 16 tasks on PromptCBLUE, 5 out-of-domain medical datasets, extensive ablations, hyperparameter sweeps, and t-SNE / routing heatmaps)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical framing, well-structured empirical comparisons, and insightful case study analyses)
  • Value: ⭐⭐⭐⭐☆ (Provides an efficient, principled blueprint for multi-task parameter-efficient fine-tuning across complex domain-specific applications)