Skip to content

CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zheng-Jay/CoMerge
Area: LLM Reasoning
Keywords: Model Merging, Multi-Task Learning, Preference Optimization, Parameter Interference, Task Vectors

TL;DR

CoMerge reformulates multi-task model merging as a self-supervised preference optimization framework, synthesizing hard negative samples from the degradation defects of naive task arithmetic merging to optimize lightweight tensor-wise coefficients (only 1,445 parameters for 8B models), achieving 0.9968 normalized performance on MergeBench while cutting GPU cost to peak by 60.1%.

Background & Motivation

Fine-tuning domain-specialized large language models has produced high-performing experts across mathematics, coding, multilingual reasoning, instruction following, and safety. Model merging has emerged as an attractive, compute-efficient paradigm for integrating these specialized capabilities into a single versatile network directly in the parameter space without retraining from scratch. However, simply performing arithmetic aggregation over task vectors induces acute parameter interference—conflicting gradient updates at shared parameter coordinates frequently degrade crucial capabilities, causing merged models to suffer catastrophic forgetting in instruction following or lose essential safety guardrails.

To alleviate parameter conflicts, existing data-free methods employ static heuristics such as pruning redundant updates and resolving sign conflicts (TIES-Merging), random drop-and-rescale operations (DARE), or geometric subspace projections (TSV-M). Data-driven strategies seek to estimate merging coefficients or localized masks by minimizing output entropy (AdaMerging), matching layer activation statistics (RegMean, RegMean++), or stitching localized regions (Localize-and-Stitch). Nevertheless, these approaches strictly optimize positive imitation or representation alignment—they define what the merged model should do, yet lack any mechanism to explicitly penalize the characteristic failure modes triggered by parameter interference, failing to learn from what the model should avoid.

This observation reveals a fundamental limitation: positive-only supervision cannot proactively steer the model away from degenerate conflict zones in the parameter space. To resolve this tension, this paper reformulates multi-task model merging into a self-supervised preference optimization problem. By treating the specialized outputs of expert models as chosen responses and the degraded, hallucinated responses of naive Task Arithmetic merging as intrinsic hard negatives, the framework establishes a bi-directional supervisory signal without requiring external annotations or reward models. Core idea: reformulate multi-task model merging as self-supervised preference optimization, synthesizing hard negative samples directly from naive merging defects and optimizing lightweight tensor-wise merging coefficients to explicitly mitigate parameter-space conflicts.

Method

Overall Architecture

The CoMerge pipeline consists of three sequential stages: multi-layer task vector extraction, self-supervised preference synthesis, and parameter-efficient preference optimization. The framework first isolates task vectors from fine-tuned experts, applying global magnitude pruning and selective low-rank SVD to filter parameter noise while freezing all backbone weights. Next, on unannotated calibration prompts, it pairs expert generations (chosen) against uncalibrated Task Arithmetic outputs (rejected) to construct the conflict preference dataset \(\mathcal{D}_{\text{pref}}\). Finally, parameterizing the fusion via learnable tensor-wise coefficients \(\Lambda\), CoMerge executes Direct Preference Optimization (DPO) to push the merged policy toward expert capability manifolds and away from naive conflict patterns with minimal GPU overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Expert Models and Pretrained Base<br/>θ_i^ft and θ_pre"] --> B["Global Magnitude Pruning & Selective SVD<br/>Retain Top-10% updates and compute SVD factors"]
    B --> C["Self-Supervised Preference Synthesis<br/>Expert output (Chosen) vs Naive Merging output (Rejected)"]
    C --> D["Tensor-wise Learnable Parameterization<br/>Formulate θ_Λ,p = θ_pre,p + Σ λ_p,i · τ̂_i,p"]
    D --> E["Parameter-Efficient Direct Preference Optimization<br/>Freeze backbone & optimize scalar coefficients Λ via DPO"]
    E --> F["Conflict-Mitigated Multi-Task Model<br/>Preserving safety, instruction, and domain reasoning"]

Key Designs

1. Global Magnitude Pruning & Selective SVD: Dual-Stage Parameter Denoising and Structural Sparsification

Directly merging raw, high-dimensional task vectors introduces substantial high-frequency noise and diffuse conflicts across unrelated coordinates. CoMerge preprocesses each task vector \(\tau_i = \theta_i^{\text{ft}} - \theta_{\text{pre}}\) through global magnitude pruning followed by selective low-rank matrix factorization. First, it retains only the globally largest 10% of parameter updates by absolute magnitude across the entire model, zeroing out minor weights that contribute negligible downstream utility. Next, for each functional projection matrix \(\bar{W}_{i,r} \in \mathbb{R}^{m_r \times n_r}\) in the attention and feed-forward blocks, it evaluates the parameter storage requirement under rank \(k_r = \min(k_{\max}, m_r, n_r)\) with \(k_{\max} = 1500\). By the Eckart-Young-Mirsky Theorem, truncated SVD provides the optimal rank-\(k_r\) approximation under the Frobenius norm: $\(\bar{W}_{i,r} \approx \hat{W}_{i,r} = \sum_{j=1}^{k_r} \sigma_j u_j v_j^\top\)$ The low-rank factorization is applied strictly when \(k_r(m_r + n_r + 1) < m_r n_r\); otherwise, the pruned projection is preserved without decomposition, and normalization weights are retained in pruned, non-factorized form. This selective factorization stabilizes the task update subspace while drastically curbing parameter redundancy.

2. Self-Supervised Preference Synthesis: Mining Endogenous Negatives from Naive Merging Defects

Existing data-driven merging techniques struggle to identify conflict boundaries because unlabeled entropy minimization or statistical feature matching cannot provide behavior-level rejection signals. CoMerge circumvents manual annotation and external reward models by exploiting the intrinsic failure modes of naive parameter aggregation. Given an unlabeled calibration query \(x \sim \mathcal{D}_{\text{cal}}\) belonging to task \(t\), the preferred output \(y_w \sim \pi_{\theta_t^{\text{ft}}}(\cdot \mid x)\) is sampled directly from expert \(\theta_t^{\text{ft}}\). Simultaneously, the rejected response \(y_l \sim \pi_{\theta_{\text{TA}}}(\cdot \mid x)\) is generated by a standard, uncalibrated Task Arithmetic model with fixed scaling \(\alpha = 0.4\). Because direct vector summation retains acute parameter interference, its outputs exhibit hallmark multi-task merging failures: hallucinated facts, reasoning breakdown, safety guardrail collapse, and stylistic incoherence. Pairing these responses into \((x, y_w, y_l)\) yields a self-supervised preference dataset \(\mathcal{D}_{\text{pref}}\) specifically targeted at penalizing the exact behavioral anomalies induced by parameter interference.

3. Tensor-wise Learnable Parameterization: Fine-Grained Conflict Resolution with Compact Parameter Space

Uniform scalar coefficients (as in Task Arithmetic) lack the expressiveness required to resolve divergent interference across different Transformer layers and sub-modules, while full-parameter fine-tuning is computationally prohibitive and prone to representation distortion. CoMerge introduces a tensor-wise coefficient parameterization where each mergeable weight tensor \(p \in \{1, \dots, P\}\) is assigned an independent scalar weight \(\lambda_{p,i}\) for each task \(i\). For Llama-3.1-8B-Instruct, indexing all attention projections, MLP layers, and normalization tensors yields \(P = 32 \times (4 + 3 + 2) + 1 = 289\) tensors; across \(N=5\) tasks, the trainable parameter matrix \(\Lambda = \{\lambda_{p,i}\} \in \mathbb{R}^{P \times N}\) comprises merely \(289 \times 5 = 1,445\) scalar values (\(1,265\) for Llama-3.2-3B and \(1,435\) for Gemma-2-2b). The merged tensor \(p\) is constructed as: $\(\theta_{\Lambda, p} = \theta_{\text{pre}, p} + \sum_{i=1}^N \lambda_{p,i} \cdot \hat{\tau}_{i,p}\)$ During optimization, all base weights and preprocessed task vectors remain strictly frozen. The model updates solely the low-dimensional matrix \(\Lambda\), establishing an inductive structural regularizer that grants granular modulation over layer-wise task contributions without risking optimization instability or catastrophic drift.

Loss & Training

The parameter optimization is guided by the Direct Preference Optimization (DPO) objective. To prevent the model from drifting excessively from the viable parameter manifold, the initial uncalibrated merged model \(\theta_{\Lambda_{\text{init}}}\) (with initialization \(\lambda_{p,i} = 0.4\)) serves as the frozen reference policy \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{pref}}(\Lambda) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_{\text{pref}}} \left[ \log \sigma \left( \beta \log \frac{\pi_{\theta_\Lambda}(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_{\theta_\Lambda}(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]\)$ with temperature \(\beta = 0.1\). Optimization is conducted using the AdamW optimizer with learning rate \(10^{-2}\), cosine learning rate scheduling, warmup ratio \(0.1\), and effective batch size \(16\) (per-device batch size \(1\) with \(16\) gradient accumulation steps) across \(280\) optimization steps. The entire procedure completes on a single NVIDIA H20 GPU.

Key Experimental Results

Main Results

Evaluated on MergeBench across 12 datasets covering five domains (Safety, Instruction Following, Mathematics, Coding, Multilingual), normalized performance (NP) benchmarks the merged model against domain-specific experts (\(1.0\) indicates perfect preservation of expert proficiency). On Llama-3.1-8B-Instruct, CoMerge outperforms all data-free and data-driven baselines:

Method Type Safety Instruction Math Coding Multilingual Avg. NP
Full-DPO Full-parameter ref. 1.0085 0.9331 1.0164 1.0521 0.9814 0.9983
Model Soup Data-free 0.9439 0.7184 0.9679 1.0082 0.9170 0.9111
Task Arithmetic Data-free 0.9917 0.8687 0.9607 1.0118 0.9363 0.9538
TIES-Merging Data-free 0.9794 0.8520 0.9577 1.0337 0.9242 0.9494
DARE Data-free 0.9456 0.7041 0.9539 1.0160 0.9208 0.9081
Dataless L&S Data-free 0.9215 0.7589 1.0251 0.9478 0.8557 0.9018
RegMean Data-driven 0.9171 0.5943 0.8445 0.7217 0.8029 0.7761
RegMean++ Data-driven 0.9530 0.8115 0.9196 0.9234 0.9448 0.9105
Localize-and-Stitch Data-driven 0.9705 0.8090 1.0095 0.9291 0.9422 0.9321
AdaMerging (Gen.) Data-driven 1.0058 0.9356 1.0007 1.0108 0.9410 0.9788
CoMerge (Ours) Data-driven 1.0340 0.9952 0.9810 1.0048 0.9687 0.9968

Ablation Study

Ablations on Llama-3.1-8B-Instruct isolate the impact of negative sample presence, negative source origin, and coefficient granularity (all variants retain global Top-10% pruning and rank-1500 SVD preprocessing):

Configuration Description Instruction Safety Avg. NP
CoMerge (Full) Full model: Tensor-wise coefficients + Task Arithmetic DPO 0.9952 1.0340 0.9968
Positive-only SFT Removes negative feedback: SFT likelihood on expert chosen samples 0.9283 1.0264 0.9842
Global coefficients Coarse parameterization: 1 shared scalar per task across all tensors 0.9857 1.0149 0.9822
Positive-only + global Dual degradation: Positive SFT with single global task scalar 0.9117 0.9993 0.9679
Base-model negatives Alternative negative source: Pretrained base model outputs as negatives 0.8138 0.9759 0.9497

In computational resource profiling to peak performance: Full-DPO demands 4 × NVIDIA H20 GPUs consuming 95.9 GPU-minutes, whereas CoMerge executes on 1 × NVIDIA H20 GPU consuming only 38.3 GPU-minutes—achieving a 60.1% reduction in GPU-resource expenditure.

Key Findings

  • Conflict-driven negative feedback is indispensable: Incorporating Task Arithmetic failure outputs via DPO improves Avg. NP from 0.9842 (Positive SFT) to 0.9968, driving a massive recovery in Instruction Following from 0.9283 to 0.9952. Swapping Task Arithmetic negatives with un-tuned base-model outputs degrades Avg. NP to 0.9497 (worse than positive-only SFT), proving that benefits originate from penalizing genuine merge conflicts rather than generic contrastive regularization.
  • Granular tensor-wise scaling is vital: Constraining optimization to global task-level coefficients drops performance to 0.9822. Layer-wise heatmaps demonstrate that instruction tasks demand higher weights in shallow-to-middle layers, whereas safety, coding, and multilingual capabilities concentrate in deeper layers.
  • Generic LLM-as-a-judge filtering harms conflict calibration: Applying DeepSeek-V3.2 in thinking mode to relabel or filter preference pairs on Gemma-2-2b-it reduces Avg. NP from 0.9338 to 0.9152 / 0.9281, demonstrating that generic output quality scores do not accurately reflect parameter-level conflict boundaries.

Highlights & Insights

  • Turning merging degradation into self-supervised preference supervision: Repurposing naive merging defects as endogenous hard negative samples elegantly solves the data acquisition bottleneck in preference alignment without requiring human annotations or external reward models.
  • Parameter efficiency with full fine-tuning performance: Optimizing merely 1,445 scalar coefficients recovers 99.68% of multi-task expert capability, performing on par with Full-DPO while shrinking compute cost by 60.1% on a single GPU.
  • Layer-depth functional specialization: Empirical coefficient inspection reveals that different tasks exhibit non-uniform layer preferences, providing valuable architectural insights for future modular routing and expert pruning.

Limitations & Future Work

  • Risk of imperfect preference label noise: The heuristic assumption that expert outputs strictly dominate naive merging outputs may occasionally invert if the expert errs or the merged model produces acceptable answers. Future research could explore confidence-weighted preference margins or domain-specific conflict judges.
  • Fixed empirical hyperparameters: The global pruning threshold (10%), maximum SVD rank (1500), and naive scaling coefficient (\(\alpha = 0.4\)) were held constant across models without systematic sensitivity exploration.
  • Diminishing returns of iterative data synthesis: Generating iterative preference rounds (D2/D3 from intermediate merged checkpoints) yielded modest gains (+0.007) and relied entirely on preserving the initial D1 dataset, indicating that dynamic self-evolving negative mining remains an open challenge.
  • vs Task Arithmetic / TIES-Merging / DARE: Static data-free methods rely on passive geometric pruning or heuristic sign resolution; CoMerge actively learns from empirical behavioral failures, using optimization to dynamically navigate away from conflict zones.
  • vs AdaMerging / RegMean++: Conventional data-driven methods depend on unilateral entropy minimization or feature covariance matching without negative feedback; CoMerge establishes a contrastive learning dynamic that explicitly instructs the model on what failure patterns to suppress.
  • vs WRPO / InfiFPO: Implicit distillation-based preference methods rely on multi-teacher rollout distillation and external reward model scoring with heavy compute overhead; CoMerge operates via explicit weight parameterization, freezing all backbone weights and preserving model topology with minimal tuning.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering framing of parameter-space merging conflicts as self-supervised DPO negative samples]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across 3 model families, 5 capabilities, 12 benchmarks, plus deep ablations and GPU profiling]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, disciplined narrative structure, and insightful empirical analysis]
  • Value: ⭐⭐⭐⭐⭐ [Highly practical and accessible framework for low-cost, high-retention multi-task LLM fusion]