Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation¶

Conference: CVPR 2026 arXiv: 2603.12577 Code: https://anonymous.4open.science/r/EPT-B0E4 Area: Robotics Keywords: [PEFT, LoRA, Mixture-of-Experts, Multi-Scale Feature Pyramid, Deconvolution Projection]

TL;DR¶

To address the limitation of MoE-LoRA methods where all experts share identical structures (uniform rank) and thus cannot adapt to tasks of varying complexity, this paper proposes EPT: a parameter pyramid constructed via a shared meta-knowledge subspace and deconvolution experts with varying kernel sizes, coupled with an Adaptive LoRA Pruner and contrastive learning-based Task Embedding. EPT achieves an average score of 87.0% on GLUE with only 0.41M parameters per task, outperforming all MoE-LoRA variants.

Background & Motivation¶

Background: PEFT, and LoRA in particular, has become the dominant paradigm for deploying LLMs to multi-task scenarios. To mitigate negative transfer caused by gradient conflicts across tasks, MoE-LoRA methods (e.g., MOELoRA, HydraLoRA, MoRE) route tokens to different low-rank experts via gating mechanisms.

Limitations of Prior Work: Nearly all existing MoE-LoRA methods employ structurally identical experts—same rank, same capacity. However, task complexity varies substantially: simple tasks (e.g., sentiment classification SST-2) require only high-level semantic abstraction, while complex tasks (e.g., linguistic acceptability judgment CoLA) demand fine-grained syntactic analysis. The authors validate this observation empirically: on different GLUE tasks with T5-base, the optimal rank varies significantly (e.g., MRPC: rank=1, RTE: rank=4, CoLA: rank=8).

Key Challenge: Uniform-architecture experts cannot capture feature granularity diversity. Low-rank experts lack expressiveness for complex tasks, while high-rank experts over-parameterize simple tasks and generalize poorly. Furthermore, each expert independently learns its own LoRA matrices, with no knowledge sharing across experts, leading to parameter redundancy.

Goal: (a) How to enable different experts to capture features at different granularities? (b) How to share common linguistic knowledge across experts while preserving task-specific adaptations? (c) How to accurately route tokens to appropriate experts?

Key Insight: Drawing inspiration from the multi-scale philosophy of Feature Pyramid Networks (FPN) in computer vision—detecting objects of different sizes requires features at different resolutions—analogously, handling NLP tasks of varying complexity requires parameter adaptations at different granularities.

Core Idea: Replace the uniform independent experts in MoE-LoRA with a parameter pyramid: deconvolution experts with different kernel sizes project from a shared low-dimensional meta-knowledge subspace to produce multi-scale weight increments.

Method¶

Overall Architecture¶

EPT replaces the LoRA modules of linear layers in Transformers. Input tokens are routed by a router to the top-k experts; each expert starts from a shared low-dimensional meta-knowledge subspace \(\mathbf{Z}_{meta}\) and projects it via deconvolution with a distinct kernel size to produce a weight increment \(\mathbf{W}_i\) at a corresponding scale. These increments are aggregated via weighted summation and added to the original pretrained weight \(\mathbf{W}_0\). The overall architecture resembles a "parameter pyramid": a compact shared meta-knowledge base at the bottom expands into multi-scale expert weights at the top.

Key Designs¶

Shared Meta-Knowledge Subspace:
- Function: Encodes common linguistic patterns across tasks, serving as the shared knowledge foundation for all experts.
- Mechanism: Defines \(\mathbf{Z}_{meta} = \mathbf{B} \cdot \mathbf{A}\), where \(\mathbf{A} \in \mathbb{R}^{R \times W_{max}}\), \(\mathbf{B} \in \mathbb{R}^{H_{max} \times R}\), and \(h, w \ll d_{model}\). Crucially, both \(\mathbf{A}\) and \(\mathbf{B}\) are initialized with random Gaussian distributions (rather than the zero initialization of standard LoRA), ensuring that \(\mathbf{Z}_{meta}\) contains rich, non-degenerate latent representations from the very beginning of training.
- Design Motivation: Conventional MoE-LoRA maintains independent LoRA matrices per expert, resulting in parameter redundancy and no knowledge sharing. EPT lets all experts share a single low-dimensional foundation; different experts merely "interpret" this foundation from different scales—analogous to multiple observers examining the same image at different resolutions.
Pyramid Projection Mechanism:
- Function: Projects the low-dimensional \(\mathbf{Z}_{meta}\) into high-dimensional weight matrices at different scales.
- Mechanism: Defines \(N\) deconvolution experts, where the \(i\)-th expert's kernel tensor \(\mathcal{K}_i\) has a distinct kernel size \(s_i\) (with stride also set to \(s_i\)). The projection is \(\mathbf{W}_i = \text{Deconv}(\mathbf{Z}_{meta}; \mathcal{K}_i)\). Experts with smaller kernel sizes capture local, fine-grained patterns, while those with larger kernel sizes capture global, long-range semantic dependencies. The implementation uses 8 experts with configurations \(\{2,2,4,4,6,6,8,8\}\).
- Design Motivation: This is the core innovation. All experts in standard MoE-LoRA share the same rank—effectively multiple copies at a single resolution. EPT creates genuine "multi-scale" diversity through deconvolution with varying kernel sizes: small-kernel experts have fewer parameters and focus on local patterns, while large-kernel experts have more parameters and cover global semantics. The kernel \(\mathcal{K}_i\) is zero-initialized to avoid perturbing the pretrained weights early in training.
- Analogy to FPN: FPN uses feature maps from different layers to capture objects at different scales; EPT uses deconvolution with different kernel sizes to extract parameter adaptations at different granularities from the same meta-knowledge.
Adaptive LoRA Pruner:
- Function: Ensures that the weight matrices output by experts at different scales strictly match the dimensions of the target pretrained layer.
- Mechanism: For a target granularity \((h_t, w_t)\), slices the first \(h_t\) rows and \(w_t\) columns from the full \(\mathbf{B}\) and \(\mathbf{A}\): \(\mathbf{Z}_{meta}^{(t)} = \mathbf{B}_{:h_t,:} \cdot \mathbf{A}_{:,:w_t}\), yielding a scale-specific meta-seed of size \(h_t \times w_t\).
- Frequency Compensation: Under uniform task sampling, shared parameters are updated at every step (frequency=1), while task-specific parameters are updated only when the corresponding task is sampled (frequency=1/T). A dimension-aware scaling factor \(d_t / T\) is introduced; the final forward pass is \(\mathbf{L} = \mathbf{W}_0 \mathbf{x} + \sum_{i \in \mathcal{P}} G(x)_i \cdot \frac{d_t}{T} \cdot (\mathbf{W}_i \mathbf{x})\), balancing gradient energy to prevent high-frequency oscillations from overwhelming the shared dimensions.
- Design Motivation: Different Transformer layers have different dimensions (e.g., attention projection vs. FFN dimensions vary considerably); the pruner ensures that meta-knowledge can flexibly adapt to any target dimension. The frequency compensation additionally resolves optimization instability arising from imbalanced multi-task sampling.
Top-k Routing + Contrastive Learning Task Embedding:
- Function: Dynamically selects the most appropriate expert combination for each token, while enhancing routing discriminability via task embeddings.
- Mechanism: Gating scores are computed as \(G(x)_i = \text{softmax}(\mathbf{W}_r \cdot x / \tau)\), and the top-k (\(k=2\)) experts are selected. A task embedding \(\mathbf{e}_t\) is learned for each task, supervised by a contrastive loss \(\mathcal{L}_{con} = -\frac{1}{M}\sum_i \log \frac{e^{s_{i,t_i}}}{\sum_k e^{s_{i,k}}}\) that pulls same-task samples closer to their task embedding while pushing apart embeddings of different tasks.
- Total Loss: \(\mathcal{L}_{total} = \mathcal{L}_{gen} + \lambda \mathcal{L}_{con}\), with \(\lambda = 0.1\).
- Design Motivation: Standard MoE routing relies solely on token features and insufficiently models inter-task relationships. Task embeddings explicitly encode inter-task correlations and distinctions via contrastive learning. PCA visualizations show that QNLI and MNLI (both NLI tasks) cluster together, while CoLA and STS-B (tasks of very different nature) are clearly separated.

Loss & Training¶

Total loss: \(\mathcal{L}_{total} = \mathcal{L}_{gen} + \lambda \mathcal{L}_{con}\), where \(\mathcal{L}_{gen}\) is the standard autoregressive generation loss, \(\mathcal{L}_{con}\) is the contrastive loss, and \(\lambda = 0.1\).
Optimizer: AdamW with peak learning rate \(3 \times 10^{-4}\), linear decay with 500-step warmup.
Training for 5 epochs, batch size 32, maximum sequence length 128.
Temperature parameter \(\tau = 0.05\) (controls the smoothness of the routing distribution).
Balanced data sampling: each task is sampled with probability \(P_t = 1/T\) to prevent large datasets from dominating training.
Re-parameterization: at inference time, the deconvolution projection results can be merged back into the pretrained weights, incurring no additional inference overhead.

Key Experimental Results¶

Main Results (GLUE Benchmark, T5-base)¶

Method	Params/Task	MNLI	QQP	QNLI	SST-2	STS-B	MRPC	RTE	CoLA	AVG
LoRA (r=8)	0.39M	85.8	89.2	93.1	93.2	90.4	89.9	76.3	62.8	85.1
MOELoRA	0.81M	86.3	90.4	93.2	94.2	89.8	90.7	79.9	65.3	86.2
MoRE	0.81M	85.6	90.2	93.1	93.9	89.9	90.7	77.7	68.7	86.2
EPT	0.41M	86.4	90.2	93.6	94.5	90.0	90.7	82.0	68.9	87.0

Commonsense Reasoning (LLaMA2-7B)¶

Method	Params/Task	BoolQ	OBQA	ARC-E	ARC-C	AVG
LoRA	2.1M	74.0	74.0	80.9	63.5	73.1
MoRE	4.5M	74.7	80.5	80.0	64.5	74.9
EPT	3.3M	76.1	78.4	81.4	66.2	75.5

Ablation Study¶

AB init	Top-K	ALP	AVG
✗	✗	✗	86.0
✗	✗	✓	86.2
✓	✗	✓	86.5
✗	✓	✓	86.7
✓	✓	✓	87.0

Each of the three components contributes approximately 0.3–0.5 points individually, with their combination yielding the best overall performance.

Pyramid Structure Comparison¶

Configuration	Description	AVG
EPT-2	All experts dim=2	86.5
EPT-4	All experts dim=4	86.2
EPT-8	All experts dim=8	86.3
EPT-2468	Mixed	87.0

The mixed multi-scale configuration consistently outperforms any uniform-scale configuration.

Highlights & Insights¶

Novel parameter pyramid concept: Transplanting the multi-scale philosophy of FPN from computer vision into the PEFT domain, constructing multi-scale experts via deconvolution with varying kernel sizes—a precise and effective analogy.
Exceptional parameter efficiency: Only 0.41M parameters per task (approximately half of MOELoRA) while achieving state-of-the-art performance. The fundamental reason is that all experts share meta-knowledge; the only independent parameters are the kernel tensors.
Re-parameterization capability: At inference time, deconvolution results can be merged back into pretrained weights with zero additional latency—a critical advantage for deployment.
Convincing expert allocation visualization: QNLI/QQP (large-dataset, complex tasks) activate high-dimensional experts 7–8, while STS-B/RTE (small-dataset, simpler tasks) activate low-dimensional experts 1–2, perfectly validating the method's core hypothesis.
Elegant frequency compensation design: The \(d_t/T\) factor balances gradient energy between shared and task-specific parameters—a widely overlooked issue in multi-task optimization.

Limitations & Future Work¶

The expert dimension configuration \(\{2,2,4,4,6,6,8,8\}\) is a static hyperparameter; dynamically searching for optimal configurations via NAS or AutoML could yield further improvements.
Evaluation is limited to NLU (GLUE + commonsense reasoning); generation tasks (e.g., summarization, translation) are not validated—PEFT behavior on generation tasks may differ.
Experiments are conducted only on T5-base and LLaMA2-7B; scalability to larger models (13B/70B) is not verified.
Contrastive learning task embeddings require task labels and are not directly applicable in task-agnostic settings (e.g., continual learning, mixed prompts).
Although the deconvolution operation involves few parameters, it still requires deconvolution computation during training; a quantitative comparison of training speed against the simple matrix multiplication of standard LoRA is absent.

vs. Standard LoRA: LoRA uses a uniform-rank BA decomposition; EPT uses multi-scale deconvolution projection. LoRA requires an independent adapter per task, while EPT enables multi-task sharing via a shared meta-knowledge and routing mechanism.
vs. MOELoRA / MoRE: These methods use multiple independent LoRA adapters as experts (identical structure, independent parameters); EPT lets all experts share a meta-knowledge foundation and project at different scales—yielding fewer parameters (0.41M vs. 0.81M) and better performance.
vs. HydraLoRA: HydraLoRA reduces redundancy by sharing the B matrix while keeping A matrices independent, but expert architectures remain uniform. EPT goes further by not only sharing the foundation but also introducing multi-scale structure.
vs. DyLoRA / AdaLoRA: These methods adapt to different tasks via dynamic rank allocation, but are limited to single-task settings and involve complex implementations. EPT naturally achieves rank adaptation in a multi-task framework via the parameter pyramid.
vs. DCFT: DCFT similarly uses deconvolution for subspace projection, but is designed for single-expert, single-task settings. EPT extends deconvolution into a multi-scale, multi-expert architecture.

Further Connections¶

Parameter pyramid → Visual PEFT: The same concept could transfer to PEFT for ViTs—different visual tasks (classification vs. detection vs. segmentation) may also require parameter adaptations at different granularities.
Meta-knowledge sharing → Federated Learning: The structure of shared \(\mathbf{Z}_{meta}\) + task-specific kernels is naturally suited for federated learning—clients only need to transmit small kernel parameters, while meta-knowledge is aggregated server-side.
Flexibility of deconvolution projection: The kernel size/stride of deconvolution can be viewed as a continuous analog to rank selection; future work could incorporate kernel parameters into the routing mechanism for finer-grained capacity allocation.

Rating¶

Novelty: ⭐⭐⭐⭐ The FPN→PEFT analogy is elegant, and the deconvolution-based parameter pyramid is a novel design, though individual components are not entirely new in isolation.
Experimental Thoroughness: ⭐⭐⭐⭐ Covers 8 GLUE tasks + 4 commonsense reasoning tasks + comprehensive ablations + expert allocation visualizations, but lacks evaluation on generation tasks and larger models.
Writing Quality: ⭐⭐⭐⭐ Motivation is clearly derived (Table 1 demonstrating task-specific optimal ranks), and the method is presented systematically.
Value: ⭐⭐⭐⭐ Achieving 87.0% average GLUE score with only 0.41M parameters is a strong result; re-parameterization with zero inference overhead makes the approach highly practical.