RUTaL: Residual Upcycling with Task Ladder for Efficient Multi-Task Learning¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Model Compression
Keywords: Multi-Task Learning / Parameter-Efficient Fine-Tuning / Model Upcycling / Mixture of Experts / Decoupled Adaptation
TL;DR¶
Addressing the prohibitive resource overhead of full fine-tuning and the representational competition inherent in coupled PEFT methods where shared and task-specific parameters contest a fixed backbone capacity, RUTaL introduces a structurally decoupled multi-task adaptation framework: it expands the shared backbone into a task-general MoE via low-rank residual upcycling (TGRU) while extracting task-specialized representations through a non-interfering side ladder branch (TSLA), achieving a state-of-the-art +7.79% overall improvement (\(\Delta m\)) on PASCAL-Context with merely 2.93M trainable parameters.
Background & Motivation¶
Pretrained vision Transformers (ViTs) and large-scale vision foundation models have become the standard initialization across dense scene understanding. When adapting these high-capacity models to multi-task learning (MTL)βwhere heterogeneous tasks such as semantic segmentation, human part segmentation, surface normal estimation, and depth estimation must be solved simultaneouslyβtraditional full fine-tuning incurs immense computational and memory overhead. Moreover, co-optimizing disparate task objectives across all backbone parameters often exacerbates negative transfer, disrupting general visual representations. Conversely, freezing the backbone and updating only task-specific decoders severely limits model expressiveness, leading to sharp multi-task performance degradation.
Although parameter-efficient fine-tuning (PEFT) techniques have emerged to mitigate adaptation costs, most early paradigms (e.g., LoRA, Adapters, Visual Prompt Tuning) were tailored for single-task adaptation. Naively deploying independent adapter modules for each task yields linear parameter expansion and overlooks cross-task synergy. On the other hand, recent multi-task PEFT frameworks (such as MTLoRA, VMT-Adapter, and TADFormer) predominantly rely on a "coupled learning paradigm": shared and task-specific adaptation parameters are co-optimized inside the fixed representational space of the pretrained backbone. Consequently, shared general knowledge and task-specialized updates compete directly for constrained backbone capacity, inducing representational interference and hindering effective multi-task capacity scaling.
The angle of attack in this work is to establish a capacity-aware, structurally decoupled adaptation principle: rather than forcing shared and task-specific updates to contest the same representational channels, task-generic capacity expansion and task-specific specialization should follow separate architectural pathways. The core idea is to introduce RUTaL, a decoupled PEFT framework that first transforms pretrained ViT feed-forward layers into a task-generic MoE backbone via low-rank residual upcycling (TGRU) to expand shared representational capacity, and subsequently extracts task-specialized features through a lightweight side ladder branch (TSLA) without perturbing the shared backbone.
Method¶
Overall Architecture¶
RUTaL adopts a structurally decoupled dual-pathway architecture for multi-task dense scene understanding. The shared central backbone processes input tokens through a single forward pass, expanding model capacity and producing high-quality task-agnostic representations via low-rank residual MoE layers. Concurrently, a parallel, lightweight task ladder branch queries task-relevant features stage by stage: Task-Aware Selectors (TAS) project the backbone features into task-conditioned key-value representations and modulation vectors, which are then integrated by the Ladder Adaptation Extractor (LAE) using sparse cross-attention and affine-modulated feed-forward transformations. Finally, the hierarchical task representations from the ladder branch are fed into task-specific SegFormer MLP decoders for multi-task dense predictions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image x"] --> B["Task-General Residual Upcycling<br/>Low-Rank Residual Reparameterized MoE Backbone"]
B -->|Stage Features Fi| C["Task-Aware Selector<br/>Lightweight MLPs generate Zti and modulation Ξ³, Ξ²"]
C -->|Task Key-Values & Modulation| D["Ladder Adaptation Extractor<br/>Cross-Attention Fusion & Affine Modulated FFN"]
D --> E["Task-Specific Dense Outputs<br/>Lightweight SegFormer Decoders for Multi-Task Prediction"]
Key Designs¶
1. Task-General Residual Upcycling: Low-Rank Residual MoE with Shared Up-Projection
Addressing the fundamental limitation that a fixed-capacity backbone forces multi-task representations into mutual competition, conventional model upcycling replicates FFN weights across \(N\) experts and trains them end-to-end. However, storing full weights for all \(N\) experts incurs a steep parameter cost (\(N \times 2HD\)). TGRU reparameterizes each expert weight as the sum of the frozen base weight \(W_{\text{base}}\) and a low-rank residual increment \(\Delta W_i = A_i B_i\), where the rank \(r \ll \min(D, H)\). Furthermore, recognizing that independent experts perform isolated forward passes without sharing fundamental knowledge, TGRU enforces a single shared up-projection residual matrix \(\Delta W_{\text{shared}}^{\text{up}}\) across all experts while retaining expert-specific residual down-projections \(\Delta W_i^{\text{down}}\). The entire MoE layer is thus functionally reparameterized into two aggregated matrices: $\(W_{\text{rep}}^{\text{up}} = W_{\text{base}}^{\text{up}} + \Delta W_{\text{shared}}^{\text{up}}, \quad W_{\text{rep}}^{\text{down}} = \sum_{n=1}^N \mathrm{Softmax}(R(\mathbf{x}))_n \left(W_{\text{base}}^{\text{down}} + \Delta W_n^{\text{down}}\right)\)$ The output is computed compactly as \(y = W_{\text{rep}}^{\text{down}} \sigma(W_{\text{rep}}^{\text{up}} \mathbf{x})\). This design shrinks trainable expert parameters from \(N \cdot 2r(D+H)\) to \((N+1) \cdot r(D+H)\), facilitating seamless cross-expert knowledge transfer and expanding the shared representational space in a single forward pass.
2. Task-Aware Selector: Decoupled Key-Value Generation and Task Conditioning
After the upcycled backbone extracts enriched task-general representations, distinct downstream tasks require completely different feature components (e.g., boundary delineations for semantic segmentation versus continuous surface normals). Introducing task-specific routing within the backbone would contaminate shared representations through cross-task gradient conflicts. RUTaL therefore decouples task specialization into dedicated TAS modules located in the side pathway. For each task \(t \in \mathcal{T}\) at stage \(i\), TAS deploys two lightweight two-layer MLPs: the Task Feature Projector \(P^t(\cdot)\) maps backbone features \(F_i \in \mathbb{R}^{L \times D}\) into low-dimensional task key-values \(\mathcal{Z}_i^t = P^t(F_i) \in \mathbb{R}^{L \times \hat{D}}\) (where \(\hat{D} = D/4\)); concurrently, the Task Modulation Generator \(R^t(\cdot)\) regresses task-specific affine modulation vectors \(\{\gamma_i^t, \beta_i^t\} \in \mathbb{R}^{L \times \hat{D}}\). By isolating task-dependent parameters strictly within these lightweight projectors, task-specific conditioning is achieved with minimal parameter footprint.
3. Ladder Adaptation Extractor: Shared Side Pathway with Task-Conditioned Cross-Attention and Modulation
To progressively accumulate multi-scale task representations without propagating gradients back into the frozen backbone, TSLA constructs a shared auxiliary ladder pathway. The LAE modules share identical parameters across all tasks, receiving their task specificity entirely from the dynamic inputs generated by TAS. At stage \(i\), LAE takes the previous stage feature \(\mathcal{A}_{i-1}^t\) (with \(\mathcal{A}_0^t\) initialized from the patch embedding), linearly projects it to \(\hat{\mathcal{A}}_{i-1}^t\), and injects task-adapted key-values \(\mathcal{Z}_i^t\) via deformable sparse cross-attention to produce fused representation \(\mathcal{H}_{i-1}^t\). Subsequently, a task-modulated FFN conditions the intermediate features using affine scaling and shifting: $\(\hat{\mathcal{A}}_i^t = W^{\text{down}} \left( \gamma_i^t \odot \sigma(W^{\text{up}} \mathrm{norm}(\mathcal{H}_{i-1}^t)) + \beta_i^t \right)\)$ A final projection restores the channel dimension to yield \(\mathcal{A}_i^t\) for subsequent stages. This interplay of cross-attention querying and affine modulation enables robust task specialization while maintaining task isolation.
A Worked Example¶
Consider a Swin-Tiny backbone taking a \(448 \times 448\) image to execute 4 dense prediction tasks simultaneously (semantic segmentation, human parts, surface normals, and saliency): 1. Backbone Upcycling: In all 4 stages of Swin-Tiny, FFN layers operate as \(N=4\) expert MoEs with rank \(r=8\). During a single forward pass, the shared up-projection and dynamic softmax-gated down-projection process tokens collectively, yielding multi-scale general representations \(F_1, F_2, F_3, F_4\). 2. Task-Specific Filtering: At Stage 2 (channel dimension \(D=192\)), 4 separate task-specific TAS projectors map \(F_2\) to compressed \(\hat{D}=48\) dimension key-values \(\mathcal{Z}_2^t\) and modulation pairs \((\gamma_2^t, \beta_2^t)\). Saliency detection isolates foreground cues, while surface normal estimation extracts fine geometric orientations. 3. Ladder Feature Fusion: The shared LAE module utilizes 6 attention heads and 4 deformable sampling points to inject \(\mathcal{Z}_2^t\) into the running ladder state \(\mathcal{A}_1^t\), followed by element-wise scaling by \(\gamma_2^t\) and offset by \(\beta_2^t\), producing stage output \(\mathcal{A}_2^t\). 4. Decoded Prediction: The 4-stage ladder representations \(\{\mathcal{A}_i^t\}_{i=1}^4\) feed into separate SegFormer MLP decoders, generating high-resolution predictions for all tasks in parallel.
Loss & Training¶
The entire architecture is trained using a weighted multi-task objective: $\(\mathcal{L} = \sum_{t \in \mathcal{T}} \omega_t \mathcal{L}_t\)$ where semantic segmentation and human part segmentation use standard cross-entropy loss, surface normal estimation uses \(L_1\) loss, and saliency detection employs balanced cross-entropy loss. Task loss weights \(\omega_t\) follow standard multi-task benchmarks (matching MTLoRA and TADFormer). Training is conducted using AdamW for 300 epochs with an initial learning rate of \(4 \times 10^{-4}\), weight decay of \(1 \times 10^{-4}\), and a cosine learning rate decay schedule. Pretrained backbone base weights remain frozen throughout adaptation; only the low-rank residual matrices, routing networks, TSLA ladder modules, and SegFormer heads are updated.
Key Experimental Results¶
Main Results¶
On the PASCAL-Context benchmark (evaluating semantic segmentation, human part segmentation, saliency detection, and surface normal estimation) with a Swin-Tiny backbone, RUTaL demonstrates clear superiority across dense multi-task metrics:
| Category | Method | SemSeg (mIoU β) | HumPa (mIoU β) | Saliency (mIoU β) | Normals (RMSE β) | Overall Gain \(\Delta m\) (%) β | Trainable Params #TP (M) β |
|---|---|---|---|---|---|---|---|
| Traditional Fine-Tuning | Single-task Full FT | 67.21 | 61.93 | 62.35 | 17.97 | 0.00 | 112.62 |
| Multi-task Full FT | 67.56 | 60.24 | 65.21 | 16.64 | +2.23 | 30.06 | |
| Decoder-only FT | 65.09 | 53.48 | 57.46 | 20.69 | -9.95 | 1.94 | |
| Single-Task PEFT | Adapter (ICLR 22) | 69.21 | 57.38 | 61.28 | 18.83 | -2.71 | 11.24 |
| VL-Adapter (CVPR 22) | 70.21 | 59.15 | 62.29 | 19.26 | -1.83 | 4.74 | |
| LoRA (ICLR 22) | 70.12 | 57.73 | 61.90 | 18.96 | -2.17 | 2.87 | |
| BitFit (ACL 22) | 68.57 | 55.99 | 60.64 | 19.42 | -4.60 | 2.85 | |
| Multi-Task PEFT | HyperFormer (ACL 21) | 71.43 | 60.73 | 65.54 | 17.77 | +2.64 | 75.32 |
| Polyhistor (NeurIPS 22) | 70.87 | 59.54 | 65.47 | 17.47 | +2.34 | 8.96 | |
| VMT-Adapter (AAAI 24) | 71.60 | 60.67 | 64.02 | 16.41 | +3.96 | 3.68 | |
| MTLoRA (CVPR 24) | 67.90 | 59.84 | 65.40 | 16.60 | +2.55 | 8.34 | |
| DiTask (CVPR 25) | 70.09 | 59.03 | 64.55 | 17.47 | +1.48 | 3.55 | |
| TADFormer (CVPR 25) | 70.82 | 60.45 | 65.88 | 16.48 | +4.24 | 7.38 | |
| Ours | RUTaL | 73.42 | 61.70 | 64.74 | 14.65 | +7.79 | 2.93 |
On the NYUDv2 indoor scene understanding benchmark (surface normals, depth, and semantic segmentation):
| Method | Normals (RMSE β) | Depth (RMSE β) | SemSeg (mIoU β) | Trainable Params #TP (M) β |
|---|---|---|---|---|
| MTLoRA (CVPR 24) | 27.38 | 0.6791 | 36.33 | 7.81 |
| TADFormer (CVPR 25) | 27.16 | 0.6577 | 37.30 | 6.47 |
| RUTaL (Ours) | 21.75 | 0.6175 | 42.76 | 2.58 |
In terms of computational and memory efficiency on PASCAL-Context: - Training Batch Time: RUTaL requires only 0.248 s/batch, outperforming MTLoRA (0.305 s/batch) and TADFormer (0.458 s/batch). - GPU Memory: RUTaL consumes 12.67 GB, cutting 7.80 GB compared to TADFormer (20.47 GB) and 5.55 GB compared to MTLoRA (18.22 GB).
Ablation Study¶
Ablations on module combinations, upcycling strategies, and TSLA sub-components on PASCAL-Context validate every design choice:
| Module Combination | TGRU | TSLA | Trainable Params #TP (M) | Overall Gain \(\Delta m\) (%) | Note |
|---|---|---|---|---|---|
| Backbone Upcycling Only | β | β | 1.76 | +2.42 | Lacks task-specific branch; MoE features shared directly |
| Side Ladder Only | β | β | 2.02 | +6.75 | Backbone capacity unexpanded; queries from frozen ViT |
| Full Model | β | β | 2.93 | +7.79 | Decoupled expansion and specialization work synergistically |
Comparison of different Upcycling strategies in TGRU:
| Upcycling Strategy | Representation | Expert Param Scale | Extra #TP (M) | \(\Delta m\) (%) | Empirical Analysis |
|---|---|---|---|---|---|
| Fully Fine-tuned MoE | Full weights \(W_i\) | \(N \times 2\) | 71.13 | +4.72 | Router overfits on large parameter space; severe redundancy |
| Per-Expert Low-Rank | Low-rank \(\Delta W_i\) | \(N \times 2\) | 3.45 | +7.60 | Lacks cross-expert parameter sharing |
| Reparameterized Aggregation | Shared Up + Gated Down \(W_{\text{rep}}\) | \(N + 1\) | 2.93 | +7.79 | Optimal regularization, knowledge transfer, and efficiency |
Ablation of task adaptation components within TSLA: - Feature Projector \(P^t\) with Cross-Attention (CA) alone: #TP = 2.52M, \(\Delta m = +6.71\%\); - Modulation Generator \(R^t\) with Task Modulation (TM) alone: #TP = 2.55M, \(\Delta m = +3.01\%\); - Combined Projector & Modulation (\(P^t\) & \(R^t\), CA & TM): #TP = 2.93M, \(\Delta m = +7.79\%\).
Key Findings¶
- Decoupled Pathways Prevent Negative Transfer: Single-task PEFT methods suffer severe negative transfer in multi-task scenarios (e.g., LoRA yields \(\Delta m = -2.17\%\)). In contrast, establishing the decoupled TSLA side branch alone turns the gain to \(+6.75\%\), proving that isolating task specialization from shared representations is critical.
- Low-Rank Reparameterization Regularizes MoE Upcycling: Scaling MoE upcycling to full-rank tuning introduces 71.13M parameters but achieves only \(+4.72\%\) due to router collapse in dense prediction tasks. The low-rank residual reparameterization provides strong implicit regularization, attaining \(+7.79\%\) with only 2.93M parameters.
- Consistent Scaling Across Vision Foundation Models: On the DINOv2-Reg (ViT) foundation model, RUTaL attains \(\Delta m = +15.72\%\) with 28.98M parameters, surpassing both multi-task full fine-tuning (+15.70%, 113.68M) and MTLoRA (+14.57%, 34.15M), validating its cross-architecture scalability.
Highlights & Insights¶
- Dual-Pathway Decoupled Architecture: By bifurcating multi-task requirements into shared backbone capacity expansion and decoupled ladder feature extraction, the model eliminates parameter contention between generic visual semantics and specialized task needs.
- Shared Up-Projection MoE Reparameterization: Merging the up-projection residual across all experts into a single shared matrix \(\Delta W_{\text{shared}}^{\text{up}}\) halves the MoE residual parameters while enforcing cross-expert knowledge distillation, enabling high-capacity representation in a single forward pass.
- Practical Memory & Compute Efficiency: By retaining large-scale computations within the frozen backbone and sharing the ladder extractor across tasks, RUTaL lowers GPU memory footprint to 12.67 GB and training batch latency to 0.248 s, offering an ideal recipe for resource-constrained deployment.
Limitations & Future Work¶
- Task Domain Scope: Empirical evaluations are currently restricted to multi-task dense visual perception; extending this decoupled paradigm to vision-language multi-task reasoning or multi-modal generative agents remains unaddressed.
- Linear Scaling of Selectors: Although TAS MLPs are lightweight, their parameter count still scales linearly \(O(K)\) with task count \(K\). In scenarios with hundreds of downstream tasks, sharing meta-networks across tasks could further improve scalability.
- Future Directions: Exploring hypernetwork-generated TAS modules to enable zero-shot task addition and investigating RUTaL's effectiveness in continual multi-task learning to prevent catastrophic forgetting.
Related Work & Insights¶
- vs MTLoRA (CVPR 24): MTLoRA decomposes LoRA into task-agnostic and task-specific matrices inside the frozen backbone, but operates within the constrained backbone space. RUTaL expands backbone capacity via MoE upcycling and queries via an external ladder, outperforming MTLoRA (+7.79% vs +2.55%) with less than 40% of its parameters (2.93M vs 8.34M).
- vs TADFormer (CVPR 25): TADFormer relies on dynamic input-conditioned Transformers, which introduce high computational latency (0.458 s/batch) and 7.38M parameters. RUTaL cuts training time nearly in half (0.248 s/batch), reduces memory by 7.80 GB, and improves overall accuracy by 3.55%.
- vs Sparse Model Upcycling: Traditional upcycling blindly duplicates dense weights and fine-tunes all parameters. RUTaL demonstrates that low-rank residual decomposition combined with shared up-projection provides superior regularization and parameter efficiency for multi-task adaptation.
Rating¶
- Novelty: βββββ Decoupled multi-task PEFT paradigm integrating low-rank residual MoE upcycling with an external task ladder branch.
- Experimental Thoroughness: βββββ Comprehensive evaluations across PASCAL-Context, NYUDv2, and DINOv2 backbones with extensive ablations and throughput/memory profiling.
- Writing Quality: βββββ Clear problem formulation, intuitive diagrams, and rigorous mathematical formulations.
- Value: βββββ Highly practical framework for deploying multi-task vision foundation models under strict compute and memory budgets.