Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging¶

Conference: ECCV2026
Paper: https://eccv.ecva.net/virtual/2026/poster/4368
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/5605.pdf
Authors: Kuangpu Guo, Aijing Yu, Jian Liang, Yuhe Ding, Zilei Wang, Ran He, Tieniu Tan
Code: https://github.com/krumpguo/DTS
Area: Model Compression
Keywords: model merging, low-rank approximation, four-group thresholding, task personalization, unseen-task generalization

TL;DR¶

Rather than forcing every task to use identical weights, DTS stores reconstructable task residuals through low-rank decomposition and four-group thresholding, achieving 90.17% average accuracy in eight-task ViT-B/32 merging with 0.98% additional storage per task and combining residuals by task semantic similarity for unseen tasks.

Background & Motivation¶

Fine-tuning a common pretrained model separately for different tasks produces strong experts, but deploying them requires storing many complete models. Weight-Averaging, Task-Arithmetic, and Ties-Merging attempt to consolidate these experts into shared weights, where task-specific parameter updates can interfere. The paper first examines pairwise merging: even merging an SVHN expert with the semantically related MNIST expert reduces SVHN performance. The issue is therefore not limited to highly dissimilar tasks; a single parameter state can lose details needed by different specialized capabilities.

Personalized model merging acknowledges these differences by retaining task-specific components and activating the relevant information according to task identity. EMR-Merging uses modulators, T-Switch stores binarized task vectors, and WEMOE retains relatively large expert components. These approaches alleviate interference but reintroduce storage costs. This paper focuses on encoding task differences rather than adding a new multi-task training objective: if an expert's deviation from a shared base can be approximated with very few bits, most storage can be shared while preserving performance close to individual models.

A further difficulty is that a new task has no expert residual of its own. Simply reverting to the shared model wastes information in existing experts, while training a router requires additional data. The paper uses class names or task descriptions as semantic signals available offline to assign transfer weights to existing residuals. Core idea: represent personalization as a residual relative to a reference model, compress it with a low-rank approximation, then encode sign and magnitude groups with scale recovery; reconstruct individual weights for seen tasks and mix existing residuals by semantic similarity for unseen tasks.

Method¶

Overall Architecture¶

The inputs are fine-tuned models with compatible parameter structures and a reference model. DTS first identifies the task differences to retain through “Reference Residuals,” reduces their representation through “Low-Rank Decomposition,” and compresses singular vectors through “Four-Group Encoding and Scale Recovery.” Before inference, it adds the approximated residual back to the reference model to obtain complete weights for the current task.

Seen tasks select their own residual using a task ID, without running a router for every sample. For unseen tasks, “Semantic Residual Transfer” mixes residuals according to text-based similarity between the target and seen tasks. This is therefore a personalized scheme with a shared base and lightweight task states, not conventional merging in which every input uses the same fixed weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Fine-tuned and reference models"] --> B["Reference Residuals"]
    B --> C["Low-Rank Decomposition"]
    C --> D["Four-Group Encoding and Scale Recovery"]
    D -->|Seen task ID| F["Add to reference model<br/>Cache complete task weights"]
    D -->|Unseen task| E["Semantic Residual Transfer"]
    E --> F

Key Designs¶

1. Reference Residuals: separate expert personalization from the shared base

DTS-T uses task vectors, defined as fine-tuned weights minus the original pretrained weights. DTS-D uses difference vectors, defined as fine-tuned weights minus the weights of a basic merged model. Both representations use the same compression pipeline, but reconstruction must add each residual back to its corresponding base; they are not interchangeable. The base for DTS-D can be produced by a method such as Ties-Merging, and its residual expresses what still needs to be corrected in the shared model for a particular expert.

This distinction also affects availability requirements. DTS-T requires the original pretrained parameters; DTS-D does not require them when a merged model is already available. The latter provides a starting point containing shared knowledge for unseen-task transfer. It does not, however, imply that models with arbitrary architectures or parameter permutations can be subtracted. The experiments follow the setting of merging multiple fine-tuned models built on the same backbone.

2. Low-Rank Decomposition: prioritize the dominant matrix directions in task differences

For each two-dimensional residual matrix, DTS performs truncated singular value decomposition and retains a subset of the largest singular values and their left and right singular vectors. The paper uses \(r\) for the fraction of singular values retained, not an integer rank; thus, \(r=0.3\) retains approximately thirty percent of the singular directions. Higher-dimensional parameters such as convolutional kernels are reshaped into matrices, whereas one-dimensional parameters such as biases bypass decomposition and proceed directly to encoding. The retained singular values must also be stored; the representation is not just binary masks.

SVD is motivated by its optimality for fixed-rank matrix approximation, allowing a few directions to summarize major residual changes. This result concerns matrix reconstruction error, however, and does not guarantee that every discarded direction is irrelevant to the task or that accuracy is optimal. The paper makes \(r\) a storage-budget control: tighter budgets retain fewer directions without changing the overall procedure.

3. Four-Group Encoding and Scale Recovery: distinguish directions with two bits, then restore their magnitudes

Low-rank factors are still floating-point matrices, so directly storing them is not sufficiently compact. DTS first separates singular-vector elements by sign, then splits the positive and negative subsets at their respective medians into larger and smaller groups. Each element therefore belongs to one of four groups: one bit identifies its sign, and another identifies its magnitude level. Compared with sign-only binarization, this retains coarse strength differences among elements of the same sign instead of assigning all positive or negative elements one common scale.

Each group stores an additional scale equal to the arithmetic mean of its elements' absolute values, with the group sign restored during reconstruction. The paper uses the \(L_1\) norm to calculate the sum of absolute values divided by group size; it does not select a median as the minimizer of an \(L_1\) loss. For a fixed partition, the signed group mean is the constant approximation minimizing within-group squared error. With \(S_k\) denoting a same-sign group, \(x_j\) an element, and \(\sigma_k\) its group sign, the prose rule can be written compactly as:

\[ s_k=\frac{1}{|S_k|}\sum_{j\in S_k}|x_j|,\qquad \hat{x}_j=\sigma_k s_k\quad (j\in S_k). \]

This is equivalent notation organized from the paper's prose, not a verbatim transcription of the damaged cached equation. After applying the procedure to both left and right singular vectors, reconstruction follows:

\[ \hat{\theta}_n=\theta_{\mathrm{ref}}+\hat{U}_n\Sigma_n\hat{V}_n^{\mathsf T}. \]

The reference weights are the pretrained model for DTS-T or the basic merged model for DTS-D. This also explains why scale recovery is indispensable: signs and groups retain patterns, but without the actual magnitudes, the reconstructed update may be severely distorted. The compression associated with two-bit encoding applies only to encoded elements. Total per-task overhead also includes singular values, group scales, and other parameters, so the entire model cannot simply be described as two-bit.

4. Semantic Residual Transfer: combine existing differences using task descriptions when no target expert exists

Classification tasks are represented by class names: a pretrained text encoder embeds each name, and their mean forms the task embedding. For generation tasks, the paper proposes language representations of task instructions or dataset descriptions. Cosine similarities between the target embedding and each seen-task embedding are normalized by their sum to obtain coefficients for combining existing approximated residuals. This requires neither target-task training samples nor a newly learned router, but it still requires the target classes or task description.

For DTS-D, \(E_u\) and \(E_n\) denote the semantic embeddings of the unseen and seen tasks, and \(\hat{d}_n\) is a stored approximate difference vector. Following the prose in Section 3.3 and the meaning of Eq. (8), the combination is:

\[ \gamma_n=\frac{\cos(E_u,E_n)}{\sum_{k=1}^{N}\cos(E_u,E_k)},\qquad \hat{\theta}_u=\theta_m+\sum_{n=1}^{N}\gamma_n\hat{d}_n. \]

DTS-T substitutes approximate task vectors and the pretrained model for difference vectors and the basic merged model. These are not softmax weights: the paper does not specify handling for negative similarities or a near-zero denominator, so temperature, clipping, or stabilization terms should not be silently added. Semantic proximity is a proxy for parameter transferability, not a guarantee of task compatibility.

Loss & Training¶

DTS adds no gradient-based training and learns no new residual loss. Its offline stage computes residuals, SVD factors, groups, and mean scales from existing checkpoints. These operations compress existing experts rather than replace their original fine-tuning. The default is \(r=0.3\); the starred DTS-T and DTS-D variants adjust \(r\) by backbone to keep additional storage below 1% per task. The best score of a default variant and the lowest storage of a starred variant must not be presented as one configuration.

Task semantic embeddings can also be computed offline. At a task switch, complete weights are reconstructed once and cached for subsequent forward passes, so continued operation on the same task adds no new computation branch. This does not make switching free or eliminate the need to hold full runtime weights. Table 9 reports 1.06 ms for ViT-B/32 reconstruction versus 1.01 ms for rank-1 LoRA. This comparison concerns switching cost, not end-to-end latency for each input sample.

Key Experimental Results¶

Main Results¶

Conventional merging covers eight visual classification tasks with ViT-B/32 and ViT-L/14, thirty tasks with ViT-B/16, and language tasks with RoBERTa, GPT-2, and Qwen-14B. The table below selects representative results verifiable in the main paper. Avg. denotes mean task accuracy in the visual experiments. For RoBERTa and Qwen-14B, it preserves the reported mean task scores without treating every benchmark score as the same accuracy metric.

AMR is additional storage per task as a percentage of the storage of one base model, not the combined overhead for all tasks. Storage accumulates as more task residuals are retained. ADR denotes the relative performance decrease from individually fine-tuned experts, with lower values preferred. Since some reported ADR entries are inconsistent with the displayed averages, the table lists average scores and AMR directly and does not derive additional ADR values.

Backbone / Setting	Method	Avg. ↑	AMR / Task (%) ↓
ViT-B/32 / Eight tasks	Individual	90.69	—
ViT-B/32 / Eight tasks	T-Switch	90.15	6.25
ViT-B/32 / Eight tasks	DTS-D	90.40	3.68
ViT-B/32 / Eight tasks	DTS-D*	90.17	0.98
ViT-L/14 / Eight tasks	T-Switch	94.16	6.25
ViT-L/14 / Eight tasks	DTS-D*	94.14	0.99
RoBERTa / Eight tasks	T-Switch	84.33	6.25
RoBERTa / Eight tasks	DTS-D*	84.75	0.88
Qwen-14B / Three tasks	FREE-Merging	71.52	10.00
Qwen-14B / Three tasks	DTS-D*	71.70	0.92

Sources: main-paper Tables 1–4. On ViT-B/32, DTS-D* exceeds T-Switch by 0.02 percentage points while reducing task storage. On ViT-L/14, it is 0.02 percentage points lower, so higher accuracy cannot be claimed for every low-budget configuration. Individual refers to separately stored experts, not a merging method under the same budget.

Unseen-task experiments use a different protocol. ViT-B/32 merges only six seen-task experts and holds out RESISC45 and SVHN. GPT-2 merges experts for CoLA, MNLI, MRPC, and QNLI while holding out QQP, RTE, and SST-2. Target-task experts do not participate in merging, so these results are not directly comparable to conventional eight-task merging.

Backbone / Unseen-Task Protocol	Method	Unseen-Task Avg. ↑
ViT-B/32 / Two held-out tasks	Ties-Merging	53.85
ViT-B/32 / Two held-out tasks	AdaMerging	54.74
ViT-B/32 / Two held-out tasks	DTS-D*	55.50
GPT-2 / Three held-out tasks	Ties-Merging	59.55
GPT-2 / Three held-out tasks	DTS-D*	60.04

Sources: main-paper Tables 5 and 6. The corresponding gains are 0.76 percentage points over AdaMerging and 0.49 percentage points over Ties-Merging. These are modest but observable improvements, not recovery to the performance of target-task experts.

Ablation Study¶

The table below uses the default \(r=0.3\) DTS-T column for ViT-B/32 in Table 8. D, T, and S denote decomposition, thresholding, and scale recovery. Its decomposition-only configuration has its own storage setting and must not be treated as a budget-matched counterpart of the separate SVD baseline in the main comparison.

Config	Avg. Acc. (%) ↑	AMR (%) ↓	Note
D only	90.57	33.73	Retains floating-point low-rank factors
T + S, without D	90.41	9.37	Encodes residuals directly
D + T + S	90.32	3.68	Full DTS-T
D + T, without S	4.98	3.69	Removes magnitude recovery

The complete method does not have the highest accuracy in this ablation; it substantially reduces storage while retaining similar performance. Removing scale recovery reduces accuracy from 90.32% to 4.98%, showing that the encoding must preserve the numerical scale of parameter updates.

Key Findings¶

The principal benefit is smaller task states, not a large improvement over individual experts. Default DTS-D and low-budget DTS-D* should be assessed separately: the former spends more storage to remain closer to expert performance.
Decomposition and thresholding reduce different costs: one reduces retained matrix directions, while the other reduces bits per factor element. Scale recovery determines whether this discrete representation maps back to meaningful parameters.
Unseen-task gains involve both residuals and semantic weights. The advantages in Tables 5 and 6 do not solve target-domain adaptation, and “no training samples” must not be confused with “no target-task information.”

Highlights & Insights¶

Replacing complete experts with compressible differences relative to a base makes personalization compatible with shared storage. DTS-D also explicitly distinguishes updates relative to pretraining from corrections relative to a merged model.
Four-group encoding is not merely an extra bit; discrete group identities cooperate with continuous scales. Group means have a fixed-partition squared-error interpretation that also explains the severe scale-removal ablation.
Using task text to weight residuals avoids training another routing model. It offers a lightweight transfer path while making available task semantics an explicit condition.

Limitations & Future Work¶

Seen-task reconstruction relies on task IDs; mixed-source inputs with unknown task identity are not directly handled by this pipeline. Switching requires reconstruction and caching, and AMR alone does not describe peak memory in concurrent serving.
The main paper does not contain the complete supplementary tables or all implementation details. The local full text includes the main paper and references but not the cited supplement. This note therefore does not present the mentioned experiments with additional thresholding groups as independently verified results.
Some equations are damaged in the cached text, so core equivalent expressions are reconstructed only from surrounding prose. ADR entries and the storage accounting for the no-decomposition ablation across different \(r\) columns also require checking against the original implementation; no further precise conclusions are inferred from them.
The paper has no explicit standalone limitations section. The task-information requirement, cosine-normalization edge cases, and serving-memory concerns above are reading-based analysis. Small low-rank error does not automatically guarantee accuracy or successful unseen-task transfer.

vs Task-Arithmetic / Ties-Merging: These methods primarily construct a single shared parameter body, whereas DTS permits different tasks to reconstruct different weights. Comparisons must consider task-identity assumptions and additional storage, not only mean scores.
vs T-Switch: Both retain lightweight task differences. DTS combines low-rank decomposition with finer sign–magnitude groups to make storage adjustable. Its advantage is primarily lower storage at similar performance, rather than improvement on every task.
vs WEMOE / Twin-Merging: These methods preserve specialization through expert components or routing, while DTS uses compressed residuals and offline semantic weights to reduce state and adaptation costs. Task-level selection consequently depends more on metadata known in advance.

Rating¶

Novelty: 4/5. The contribution combines residuals, low-rank approximation, group scales, and unseen-task transfer rather than introducing new matrix decomposition theory.
Experimental Thoroughness: 4/5. Coverage spans vision and language, multiple backbones, storage budgets, and component ablations; supplementary details and realistic serving conditions still need verification.
Writing Quality: 3/5. The central argument is clear, but some metric definitions and broad claims need care, while damaged cached equations add reproduction overhead.
Value: 4/5. Useful for multi-task checkpoint storage and known-task switching, but not an unconditional general-purpose model-merging solution.