Skip to content

Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ABgit111/purified_task_vector
Area: Model Compression
Keywords: Model Merging, task vector, context-oriented SVD, spectral rank allocation, knowledge-aware subspace

TL;DR

PAVE projects fine-tuned model parameters into a knowledge-aware subspace via context-oriented SVD guided by activation covariance matrices, pruning task-irrelevant redundant components and adaptively allocating preserved ranks across models to resolve multi-task parameter conflicts in a plug-and-play manner.

Background & Motivation

Multi-task model merging aims to synthesize capabilities from multiple domain-specific fine-tuned models into a unified network without additional training or fine-tuning costs. Task vector arithmetic has emerged as a cornerstone building block for model merging, encapsulating the downstream learning trajectory by computing the element-wise parameter shift between fine-tuned checkpoints and their common pre-trained base model (\(\Delta W = W_{\text{FT}} - W_{\text{B}}\)). Despite its conceptual simplicity and modularity, naively merging raw task vectors frequently leads to severe performance degradation and cross-task capability interference, largely stemming from task-irrelevant parameter redundancy and noise accumulated during supervised adaptation.

Existing plug-and-play conflict mitigation techniques largely operate directly on the raw parameter space. Sparsity-driven methods such as DARE employ random Dropout masks to zero out task vector elements, while other lines of work apply plain Singular Value Decomposition (plain SVD) to truncate parameter directions. However, these paradigms suffer from three fundamental limitations: first, random masking in parameter space lacks knowledge-awareness and cannot selectively distinguish between critical task features and benign noise; second, deep neural networks encode task-specific capabilities within low-dimensional parameter subspaces, rendering coordinate-wise binary masking ineffective at cleanly disentangling abilities from redundant interference; third, plain SVD reflects only the intrinsic spectral geometry of static weights without grounding them in downstream task data distributions, which easily leads to severe knowledge erasure during aggressive truncation.

Consequently, model merging fundamentally requires a mechanism to isolate task-relevant components from redundant directions within a knowledge-aware subspace. The key insight of this paper is to utilize input activation covariance matrices gathered from a small set of task-domain samples, reorienting the decomposition so that singular components are ordered strictly by their functional contribution to the target task. Core idea: purify task vectors by decomposing fine-tuned weights into a knowledge-aware subspace via Context-Oriented SVD (CO-SVD) and pruning redundant singular components, coupled with a greedy spectral rank allocation strategy that minimizes normalized activated pruning error across heterogeneous models.

Method

Overall Architecture

The PAVE (Purified TAsk VEctors) pipeline consists of three sequential phases: activation covariance collection and context-oriented decomposition, task vector purification in the knowledge-aware subspace, and cross-model global spectral rank allocation. For each of the \(K\) fine-tuned models, a small set of task training samples is propagated to capture intermediate layer activations and compute symmetric positive semi-definite covariance matrices \(C\). Linear layer weight matrices are right-multiplied by \(C\) to accentuate task-critical directions prior to SVD truncation. After retaining the top-\(r\) singular components, reconstructed weights are mapped back via \(C^{-1}\), producing the purified task vector \(\Delta W_{\text{PAVE}}\). Finally, an adaptive greedy rank allocation balances truncation budgets across all models before applying standard merging rules such as Task Arithmetic, Ties-Merging, EMR-Merging, or WUDI-Merging.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Feed Task Samples to Fine-tuned Models<br/>Collect Linear Layer Activations & Covariance C"] --> B["Knowledge-Aware Subspace Weight Reconstruction<br/>Apply CO-SVD Truncation & Right-Multiply C^-1"]
    B --> C["Spectral Rank Allocation via Normalized Error<br/>Greedy Component Pruning Across Models"]
    C --> D["Form Purified Task Vectors ΔW_PAVE<br/>Execute Downstream Model Merging Rule"]

Key Designs

1. Knowledge-Aware Subspace Weight Reconstruction: Context-Oriented Decomposition via Activation Covariance

To overcome the inability of standard SVD to reflect downstream task semantics, PAVE adopts Context-Oriented Singular Value Decomposition (CO-SVD). For any linear layer with fine-tuned weight \(W_{\text{FT}} \in \mathbb{R}^{m \times n}\), a few task samples are passed forward to record the layer input activations \(X\), yielding the activation covariance matrix \(C = X X^\top \in \mathbb{R}^{n \times n}\). Because \(C\) characterizes the empirical data distribution under the target task, projecting the parameter matrix onto \(C\) amplifies directions that actively process domain features. Singular Value Decomposition is then executed on the composite matrix \(W_{\text{FT}} C\), retaining the leading \(r\) singular values and vectors before transforming back with \(C^{-1}\):

\[W_{\text{FT}}^\dagger = \text{SVD}_r(W_{\text{FT}} C) C^{-1} = \sum_{i=1}^r \sigma_i u_i v_i^\top C^{-1}\]

The purified task vector is subsequently formulated as \(\Delta W_{\text{PAVE}} = W_{\text{FT}}^\dagger - W_{\text{B}}\). When \(C\) is singular or ill-conditioned, positive diagonal perturbations are iteratively added to guarantee numerical invertibility. Theoretically, Theorem 1 establishes local Lipschitz continuity, proving that the Frobenius discrepancy between the empirical purified weight \(W_r^\dagger(C)\) and the ideal knowledge-preserving weight \(W_r^\dagger(C^\star)\) is bounded linearly by \(\|C - C^\star\|_2\), guaranteeing robustness against sample variability.

2. Spectral Rank Allocation via Normalized Error: Discrepancy-Minimizing Cross-Model Budget Optimization

Uniformly enforcing a fixed preserved rank ratio \(\rho\) across all models severely degrades multi-task performance because individual fine-tuned checkpoints possess distinct spectral decay profiles and varying sensitivity to low-rank truncation. To achieve fair pruning, PAVE formulates a layer-wise optimization objective that minimizes the sum of normalized activated pruning errors across all \(K\) models. For layer \(l\) and model \(i\), the activated truncation error is scaled by the squared maximum singular value \((\sigma_{i, \max}^l)^2\) to eliminate absolute magnitude discrepancies across heterogeneous tasks:

\[\min_{\{r_i^l\}_{i=1}^K} \sum_{i=1}^K \sum_{j=r_i^l + 1}^{R^l} \left( \frac{\sigma_{i, j}^l}{\sigma_{i, \max}^l} \right)^2 \quad \text{s.t.} \quad \frac{1}{K R^l} \sum_{i=1}^K r_i^l \le \rho\]

where \(R^l\) denotes the full rank of layer \(l\). This objective is solved globally and efficiently via a greedy algorithm: starting from full rank for all models, the algorithm iteratively identifies the candidate model \(t\) exhibiting the minimal normalized singular value \(s_{t, r_t^l}^l = \sigma_{t, r_t^l}^l / \sigma_{t, \max}^l\) and decrements its rank \(r_t^l\) by 1, until the average preserved rank matches budget \(\rho\). A stopping threshold \(\gamma\) is introduced to cap maximum pruning depth, ensuring that models whose task knowledge decays slowly retain sufficient capacity while highly redundant models absorb the bulk of the compression budget.

Key Experimental Results

Main Results

On the 8-task GLUE benchmark using RoBERTa-base and DeBERTa-base backbones, PAVE demonstrates substantial improvements over prior baselines including DARE, RegMean, DOGE-Merging, and ISO-Merging, narrowing the performance gap to individual single-task fine-tuned models.

Backbone & Method CoLA (MCC) SST-2 (Acc) MRPC (F1) STS-B (Corr) QQP (Acc) MNLI (Acc) QNLI (Acc) RTE (Acc) GLUE Average
RoBERTa-base
Individual Fine-tuned Upper Bound 60.18 94.04 89.22 90.63 91.41 87.20 92.71 79.06 85.55
RegMean 36.67 90.60 75.74 62.68 83.55 70.02 82.35 58.48 70.01
DOGE-Merging 28.64 88.53 81.13 78.46 82.34 59.53 77.27 67.87 70.46
ISO-Merging 28.47 86.93 80.64 67.63 84.74 65.23 78.75 73.29 70.71
EMR-Merging 39.96 93.35 86.27 82.77 89.72 85.45 89.57 74.37 80.18
EMR-Merging w/ DARE 40.58 93.23 86.52 82.86 89.93 85.98 89.68 75.09 80.48 (+0.30)
EMR-Merging w/ PAVE 60.42 93.69 87.75 89.73 89.13 85.27 92.09 76.17 84.28 (+4.10)
Ties-Merging 20.48 84.40 81.13 58.19 85.70 64.65 74.81 42.96 64.04
Ties-Merging w/ PAVE 25.45 82.45 79.41 64.03 85.43 69.93 77.19 51.62 66.93 (+2.89)
WUDI-Merging 22.13 89.45 80.15 61.24 82.35 65.80 77.85 65.34 68.04
WUDI-Merging w/ PAVE 34.77 88.76 80.39 68.82 83.26 71.50 78.58 64.26 71.29 (+3.25)
DeBERTa-base
Individual Fine-tuned Upper Bound 59.10 95.06 89.21 91.31 91.50 88.52 93.30 69.31 84.66
EMR-Merging 47.05 94.72 71.32 87.57 88.36 87.38 92.29 64.26 79.11
EMR-Merging w/ DARE 47.76 94.61 72.06 87.30 88.77 87.67 92.28 66.43 79.61 (+0.50)
EMR-Merging w/ PAVE 61.32 94.50 84.31 89.54 90.34 87.50 93.01 60.65 82.64 (+3.53)

Ablation Study

To evaluate the individual contributions of the adaptive rank allocation strategy and alternative matrix decomposition operators, Table 4 and Table 5 evaluate DeBERTa on GLUE across various merging algorithms:

Configuration / Variant EMR-Merging Task Arithmetic Ties-Merging Note
Raw Merging Baseline (Unpurified) 79.11 51.01 45.95 Plain task vectors without purification
Plug-in Plain SVD 80.77 50.85 41.29 Knowledge-agnostic SVD causes severe drops on Ties (-4.66)
Plug-in ASVD 82.87 52.59 44.22 Diagonal activation magnitude scaling
Plug-in SVD-LLM 82.74 52.76 46.70 Cholesky-based whitening decomposition
PAVE plain (Uniform Rank Allocation) 81.82 52.34 48.29 Fixed preserved rank ratio across all layers and models
PAVE Full (Adaptive Spectral Allocation) 82.64 (+0.82) 52.83 (+0.49) 49.06 (+0.77) Greedy spectral rank allocation yields consistent gains

Key Findings

  • Drastic Restoration of Vulnerable Tasks: Fragile tasks highly vulnerable to parameter collision during multi-task fusion, notably CoLA, exhibit dramatic gains. Under RoBERTa, PAVE elevates CoLA from 39.96% to 60.42% (surpassing the individual expert model score of 60.18%); under DeBERTa, CoLA jumps from 47.05% to 61.32% (surpassing individual 59.10%).
  • Crucial Role of Domain Knowledge-Awareness: Replacing the covariance matrix \(C\) with activations from an unrelated task (e.g., using CoLA activations to purify SST-2 weights) or random noise triggers sharp performance collapse under low-rank truncation, proving that domain-matched activations are essential to cleanly isolate true task knowledge from parameter redundancy.
  • Robustness on Generative LLMs: When merging mathematical reasoning (WizardMath-7B) and code generation (EvolCodeAlpaca-7B) capabilities on LLaMA-2-7B, PAVE improves HumanEval score by +1.9% (reaching 35.4%) while avoiding the negative transfer and performance regression commonly observed with DARE.

Highlights & Insights

  • From Parameter Pruning to Subspace Purification: Replaces heuristic coordinate-wise Dropout masking with context-oriented spectral decomposition, enabling a geometrically principled way to isolate and prune task-irrelevant parameter shifts.
  • Scale-Normalized Spectral Rank Allocation: Normalizing the pruning loss by the maximum singular value establishes a balanced spectrum scale across disparate tasks, enabling a simple greedy routine to solve cross-model budget trade-offs with negligible computational cost.
  • Surrogate Data Compatibility: When original training corpora are inaccessible due to privacy constraints, empirical tests confirm that covariance matrices computed from related public datasets or LLM-synthesized prompt data retain over 95% of the purification efficacy.

Limitations & Future Work

  • Activation Data Dependency: Although requiring only a lightweight calibration batch (256–512 samples), PAVE is not strictly data-free, creating friction in environments with complete data access restrictions.
  • SVD Computational Overhead: Performing layer-wise SVD across wide linear projections introduces computational overhead (taking several minutes on 7B models), which is heavier than element-wise binary mask operations.
  • Generative Merging Headroom: While PAVE successfully preserves reasoning abilities during rank pruning on generative models, multi-task gains on generative benchmarks remain more modest than on classification tasks, reflecting the inherent challenge of fusing diverse autoregressive generative trajectories.
  • vs DARE: DARE applies random coordinate Dropout directly on task vectors in the parameter space, which risks disrupting low-rank task manifolds; PAVE purifies parameters along the dominant spectral directions within a knowledge-aware subspace.
  • vs RegMean / Fisher-Merging: RegMean solves a closed-form ridge regression problem on weight matrices, demanding complex matrix inversions in parameter space; PAVE acts as an orthogonal plug-and-play purification operator that seamlessly interfaces with any downstream vector merging heuristic.
  • vs DOGE-Merging / ISO-Merging: While these methods incorporate standard SVD on task vectors, their lack of activation-driven context causes severe knowledge loss when truncating singular vectors; PAVE's activation covariance weighting ensures faithful preservation of task-salient representations.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Introducing context-oriented decomposition to purify task vectors is mathematically elegant and well-grounded.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning GLUE classification, generative math/coding, and 8 vision tasks with deep ablation analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear geometric intuition, rigorous theoretical bounds, and well-structured empirical narratives.
  • Value: ⭐⭐⭐⭐⭐ A versatile plug-and-play enhancement that consistently empowers modern task vector merging techniques across architectures.