On the Vulnerability of Parameter-Level Defenses to Model Merging

Conference: ECCV 2026
arXiv: 2606.30360
Code: https://github.com/krumpguo/secure-merge-attack
Area: Model Compression / Model Merging
Keywords: Model Merging, Parameter-Space Defense, Intellectual Property Protection, Anchor Attack, Anti-Merging Fine-Tuning

TL;DR

This paper exposes a fundamental vulnerability of parameter-level defenses that use linear transformations to make models "unmergeable"—namely, the protected task vector is 2 to 3 orders of magnitude smaller than the pre-trained anchor. Consequently, this work proposes the Anchor-Guided Attack (AGA). By treating the public pre-trained model as a static anchor, AGA analytically reconstructs the hidden transformation matrix using least squares and the Hungarian algorithm, recovering the protected model almost losslessly. To counter this, an attack-resistant defense named Anchor-Repulsive Fine-tuning (ARF) is introduced as a remedy.

Background & Motivation

Merging multiple expert models fine-tuned on the same pre-trained backbone directly in the parameter space (Task Arithmetic and its successors like CAT Merging, LOT Merging, etc.) allows for constructing a multi-task model without any retraining. This highly popular paradigm of model merging is so efficient that it is practically "free": a "free rider" can simply download hard-earned fine-tuned checkpoints from HuggingFace or ModelScope, merge them with their own models, and inherit proprietary capabilities without cost, posing significant intellectual property (IP) risks. To address this issue, several "active defense" methods have emerged. Their common core idea is to silently multiply fine-tuned weights by a secret linear transformation to disrupt the alignment required for linear arithmetic between different models, thereby causing unauthorized merging to crash while ensuring model performance remains intact when used individually. Representative works include: Params, which transforms attention via diagonal matrices and disrupts MLP hidden neurons using permutation matrices; MergeLock, which locks the attention parameter space using invertible/orthogonal matrices; and MergeBarrier, which goes further by rewriting the topology of MLPs via Taylor expansion to make task vector inversion mathematically ill-posed. These methods successfully prevent naive merging attempts in experiments.

However, these defenses are built on the foundational assumption of "linear transformation," and the authors discovered a fatal flaw through a straightforward decomposition of the protected weights. Any fine-tuned weights can be written as "pre-trained weights + task vector." When the defender multiplies these by a secret matrix \(P\), the transformation is distributed across both components: \(W^p = (W_{pre}+\tau)P = W_{pre}P + \tau P\). By measuring the Frobenius norms of these two terms across multiple backbones such as ViT-B/32, the authors found that the transformed task vector \(\tau P\) is fully 2 to 3 orders of magnitude smaller than the transformed pre-trained anchor \(W_{pre}P\). In other words, the parameter space of the protected model is almost entirely dominated by the "publicly accessible" pre-trained anchor, making the task vector negligibly small. This leads to a Key Challenge that defenders fail to realize: all of their secrets are hidden in the transformation matrix \(P\), yet \(P\) acts on a quantity (pre-trained weights) that is already perfectly known to the attacker. By simply "aligning" the protected model with the public anchor, the attacker can reverse-solve \(P\), entirely compromising the security of the defense.

Core Idea: By treating the public pre-trained model as a static reference anchor, the inverse of the defense transformation matrix is analytically recovered by minimizing the discrepancy between the protected model and the anchor. Continuous least-squares closed-form solutions are used for the attention modules, and the Hungarian algorithm is used to find optimal permutations for the MLP modules. This strips off the protection and recovers the original fine-tuned weights almost losslessly, even in "defense-agnostic" scenarios where the attacker has no knowledge of which specific defense is applied.

Method

Overall Architecture

The input to AGA is a "protected fine-tuned model" \(W^p\) and a publicly downloadable "pre-trained model" \(W_{pre}\). The output is the stripped original fine-tuned weights \(W^a \approx W_{ft}\), which can then be merged normally. The backbone of this attack can be summarized in one sentence: since \(W^p \approx W_{pre}P\) (as \(\tau P\) is negligible), find a recovery matrix \(T\) such that \(W^p T = W_{pre}\). Since the anchor term dominates the optimization landscape, this alignment mathematically forces \(PT \approx I\), which implies \(T \approx P^{-1}\). Multiplying by \(T\) cancels out the protection: \(W^a = W^p T \approx (W_{pre}P + \tau P)P^{-1} = W_{ft}\). Crucially, the attacker is "defense-agnostic"—they do not know whether each model is protected or which defense is used. Therefore, AGA is applied indiscriminately to all candidate models without relying on prior knowledge of defense configurations.

Because different architectural modules undergo transformations of varying natures (continuous matrices in attention, discrete permutations in MLPs), AGA employs a dual-solver design: continuous least squares for attention, and discrete Hungarian matching for MLPs. Finally, looking at it from the defender's perspective, the authors propose ARF to artificially amplify the magnitude of the attention task vectors during fine-tuning, thereby breaking the "magnitude disparity" premise upon which AGA relies. The overall pipeline is shown below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Protected Model W^p<br/>+ Public Pre-trained Anchor W_pre"] --> B["Anchor-Dominance Observation<br/>τP is 2-3 orders of magnitude smaller than W_pre·P"]
    B --> C["Continuous Attention Attack<br/>Least-squares closed-form solution T*"]
    B --> D["Discrete MLP Attack<br/>Hungarian algorithm for optimal permutation"]
    C --> E["Recover Original Fine-tuned Weights<br/>W^a ≈ W_ft, mergeable normally"]
    D --> E
    E -.Defender Countermeasure.-> F["Anti-Merging Fine-Tuning ARF<br/>Repels anchor only on attention"]

Key Designs

1. Anchor-Dominating Observation: Negligible Task Vector, Secrets Exposed on the Anchor

This is the foundation of the paper. By decomposing the protected weight into \(W_{pre}P + \tau P\), the authors empirically verified that the norm of the transformed task vector \(\tau P\) is 2 to 3 orders of magnitude smaller than the transformed pre-trained anchor \(W_{pre}P\) (consistently holding on ViT-B/32, ViT-L/14, GPT-2, and Qwen2-7B, as shown in the supplementary material). This means that regardless of how defenders multiply the secret matrix, the protected model in parameter space is approximately equal to \(W_{pre}P\)—where \(W_{pre}\) is a public quantity fully accessible to the attacker. While the security of the defense hinges entirely on the transformation matrix \(P\), \(P\) acts directly on a known reference, making "aligning the protected model to the public anchor" a natural entry point to reverse-solve \(P\). This disparity is ubiquitous because fine-tuning essentially introduces only minor adjustments to the pre-trained weights, making task vectors inherently small; this is not an implementation flaw of a specific defense, but a geometric inevitability of the "linear protection + fine-tuning" paradigm.

2. Continuous Attention Attack: Formulating Transformation Recovery as Closed-Form Solutions of Overdetermined Linear Systems

Attention modules (Query/Key/Value/Output projections) are typically subjected to continuous invertible, orthogonal, or diagonal matrices. For such continuous transformations, the authors directly formulate finding the recovery matrix as fitting an overdetermined linear system. Although defenders often apply coupled transformations to paired modules (e.g., Q and K), attackers in reality have no knowledge of these secret configurations. Therefore, AGA decouples the problem and solves an independent least-squares problem for each projection matrix, making it more general. For any protected matrix \(W^p\) and its pre-trained anchor \(W_{pre}\), minimizing \(\|W^p T - W_{pre}\|_F^2\) and setting the derivative to zero yields the closed-form solution:

\[T^* = ((W^p)^T W^p)^{-1} (W^p)^T W_{pre}\]

Multiplying this back to the four projection matrices reconstructs the original parameter space of the entire attention mechanism in parallel. The authors further prove an error upper bound (Theorem 1): the recovery error of the attention weights \(\mathcal{E} = \|W^a - W_{ft}\|_F\) is strictly bounded by the norm of the task vector itself \(\|\tau\|_F\). The key to the proof is that the recovery error equals the orthogonal projection of the task vector onto the column space of \(W_{ft}\), denoted as \(-\Pi_{ft}\tau\), where the spectral norm of the orthogonal projection is 1. Combined with the fact that \(\tau\) is 2 to 3 orders of magnitude smaller, this theoretical maximum error is confined to an almost negligible range. Since orthogonal and non-zero diagonal matrices are subclasses of invertible matrices, this guarantee universally applies to the diagonal transformations in Params and orthogonal transformations in MergeLock.

3. Discrete MLP Attack: Formulating Permutation Recovery as Linear Sum Assignment Problems and Solving Globals via Hungarian Algorithm

Since MLP layers are interleaved with non-linear activations, defenders typically use discrete permutation matrices to shuffle the order of hidden neurons (the output of a single model remains unchanged after shuffling). Applying continuous least squares in this context to approximate the inverse permutation introduces numerical drift and accumulates error across layers because the "strictly discrete" constraint cannot be enforced. Instead, AGA models the permutation recovery as a Linear Sum Assignment Problem (LSAP). A cost matrix \(C\) is constructed, where each element \(C_{i,j}\) represents the negative cosine similarity between the \(i\)-th row of the protected first-layer MLP weights and the \(j\)-th row of the pre-trained weights. The Hungarian algorithm is then used to find the permutation matrix \(T^*\) that minimizes the total matching cost, i.e., \(T^* = \arg\min_T \sum_{i,j} C_{i,j} T_{i,j}\) (where \(T\) is constrained to be a valid permutation matrix). Once \(T^*\) is obtained, it is multiplied on the left of the first layer to restore the row order, and its transpose (since its inverse is its transpose) is applied to the next projection layer and intermediate biases to maintain input-output equivalence of the entire MLP block. The authors further present Theorem 2: as long as the task vector satisfies \(\|\tau\|_F < \tfrac{1}{2}\delta_{min}\) (where \(\delta_{min}\) is the minimum distance between the pre-trained weights and any of their non-trivial permutated states, i.e., the "permutation margin"), the Hungarian algorithm mathematically guarantees the exact output of the inverse permutation, resulting in a recovery error of strictly zero. Because pre-trained weights in deep networks are highly diverse and \(\delta_{min}\) is very large, while task vectors are natively tiny, this condition almost always holds in practical fine-tuning, completely debunking permutation-based defenses.

4. Anchor-Repulsive Fine-tuning: Repelling Attention Weights from the Anchor to Close the Amplitude Disparity

Given that the Achilles' heel of AGA is that the task vector amplitude is too small relative to the anchor, the only radical remedy for a defender is to actively amplify the task vector amplitude during fine-tuning to close this gap. An existing tuning-stage defense, MergeGuard, uses global L2 regularization to disperse weights and create task conflicts, but it has two fatal flaws: the global constraint restricts the parameter space and degrades single-model performance; furthermore, experiments show that it is still compromised by AGA across multiple datasets, as simply "dispersing weights" does not mathematically guarantee sufficient amplitude expansion. Based on this, the authors propose ARF. Instead of penalizing the entire network, ARF applies a distance-based repulsive margin penalty only to the four projection matrices of the attention mechanism, pushing them away from the pre-trained anchor until they reach a predefined safety boundary:

\[L_{total} = L_{CE} + \lambda_{dist} \sum_{\theta \in \{W_Q,W_K,W_V,W_O\}} \min\big(0,\ \rho\|\theta_{pre}\|_2 - \|\theta - \theta_{pre}\|_2\big)\]

where \(\rho\) is the expansion ratio defining the safety margin relative to the anchor amplitude, \(\lambda_{dist}\) controls the strength of the repulsive force, and \(\min(0,\cdot)\) ensures that the penalty is only applied when the distance is insufficient. The reason for targeting only attention is that the continuous inversion of attention relies precisely on minimizing the Euclidean distance (whose upper bound is \(\|\tau\|_F\), as shown in Theorem 1), making it most sensitive to distance-based repulsion. In contrast, the discrete extraction of MLP relies on scale-invariant cosine matching, rendering Euclidean amplification ineffective. Once the margin is met during fine-tuning, standard invertible matrix transformations are applied to protect the attention mechanism. Consequently, ARF represents a complete pipeline of "repulsive fine-tuning + post-hoc linear protection," which is naturally orthogonal to post-training linear defenses.

Loss & Training

On the attacker side, AGA requires no training whatsoever and is a purely analytical closed-form solution (least squares + Hungarian algorithm), making it plug-and-play for any merging algorithm. On the defender side, the training objective of ARF is detailed above, using \(\lambda_{dist}=1.0\) and \(\rho=0.05\) across all datasets. Because the repulsive force is only applied to attention while MLP undergoes standard fine-tuning, the single-model performance degradation is negligible.

Key Experimental Results

Main Results

The evaluation covers vision classification (CLIP ViT-B/32, ViT-L/14, 8 datasets), NLP understanding (GPT-2, GLUE 8 tasks), and text generation (Qwen2-7B, AlpacaEval / GSM8K / MBPP), with merging paradigms spanning Task Arithmetic, CAT Merging, and LOT Merging. The core metric is "protected task performance": when only one model is protected and merged with others, the accuracy of its corresponding task is measured to see how much is recovered by AGA.

Backbone / Merging	Defense	Unprotected Baseline	Protected Only	Post-AGA Attack	Gain
ViT-B/32 · TA	Params	67.23	5.49	66.39	↑60.90
ViT-B/32 · TA	MergeLock	67.23	5.10	66.39	↑61.29
ViT-B/32 · TA	MergeBarrier	67.23	29.32	60.80	↑31.48
ViT-B/32 · LOT	MergeLock	79.16	4.85	78.36	↑73.51
Qwen2-7B · TA	Params-D	49.10	3.07	47.79	↑44.72
GPT-2 · TA	Params	68.79	51.44	67.22	↑15.78

A representative example: Under LOT Merging, MergeLock compresses the protected task performance from 79.16% to a mere 4.85%, whereas AGA directly restores it to 78.36%, less than 1% away from the unprotected baseline. The overall utility across multiple tasks is equally striking—MergeLock degrades average multi-task utility to 4.92%, which AGA restores to 67.17%, only 0.06% shy of the unprotected baseline's 67.23%. The only defense that was not fully restored is MergeBarrier (which alters the MLP topology, forcing AGA to bypass the modified blocks and fall back to the pre-trained MLP); nonetheless, attention is still fully resolved, raising accuracy from 26.01% to 58.56%. Composite defenses (P&L, P&B, P&L&B) also failed to resist: pure parameter-level compositions were recovered to 66.39%, and only when MergeBarrier was incorporated did it drop to around 60.66%—indicating that "simply stacking multiple defenses" does not provide robust security.

Ablation Study (ARF Defense Capability and Sensitivity to \(\rho\) / \(\lambda_{dist}\), ViT-B/32)

Configuration	Merging Accuracy (lower is better)	Single-model Accuracy (higher is better)	Description
Unprotected	67.23	88.81	Reference
MergeGuard (Without Attack)	33.97	86.31	Global regularization, still high and degrades single-model
ARF (Without Attack)	4.63	88.11	Suppressed to near-random, negligible single-model loss
ARF (Under AGA Attack)	27.42	—	Still lower after attack than MergeGuard when unattacked
\(\rho=0.02 \rightarrow 0.08\)	Protection \(28.75 \rightarrow 26.36\)	Fine-tuning \(88.32 \rightarrow 87.94\)	Insensitive to hyperparameters
\(\lambda_{dist}=0.7 \rightarrow 1.3\)	Protection \(29.26 \rightarrow 26.17\)	Fine-tuning \(88.38 \rightarrow 87.01\)	Same as above, robust

Key Findings

• AGA achieves near-perfect cracking (recovering with a gap <1%) on defenses that rely on "pure parameter-space obfuscation" (Params/MergeLock). Conversely, it can only partially restore "structural rewriting" defenses (MergeBarrier)—where attention remains invertible but MLP topology rewriting leads to ill-posed inversion, defining the boundary of the attack.
• AGA is "non-destructive": even when applied to models that are not protected, the single-model accuracy remains at 88.81%, proving that it can be applied indiscriminately to all candidate models without false positives in a defense-agnostic manner.
• The success of ARF lies in its "locality": by repelling only the attention without modifying the MLPs or other layers, it thoroughly eliminates the parameter-space proximity that AGA exploits (suppressing merging accuracy to 4.63%) while keeping the single-model performance loss under 1%, significantly outperforming the globally constrained MergeGuard.

Highlights & Insights

• The decomposition of weights into "anchor + task vector" followed by norm measurement is extremely simple yet fatal. Reducing a seemingly cryptography-strength defense problem to a geometric observation where "the task vector is too small to hide secrets" is the most elegant "aha!" moment of the paper.
• The divide-and-conquer approach of the dual-solvers demonstrates strong design capability: continuous parts are resolved using least-squares closed-form solutions, and discrete permutations are resolved globally via the Hungarian algorithm. Each matches the mathematical properties of the transformation, avoiding the numerical drift of "one-size-fits-all" continuous approximations of discrete transformations.
• The two error bounds (attention \(\le\|\tau\|_F\) and MLP strictly zero when \(\|\tau\|_F < \delta_{min}/2\)) elevate the explanation of "why the attack is effective" from empirical observation to theorem-level guarantees, directing the path of defenses—to defend, one must push \(\|\tau\|_F\) up.
• ARF's concept of "applying distance-based repulsion only where distance-based attacks can succeed, while leaving other parts untouched" is a transferable defense paradigm: identify the specific geometric quantities holding the attack's key, and perform minimal-intrusive intervention to avoid the "lose-lose" outcomes of global regularization.

Limitations & Future Work

• AGA can only bypass rather than reverse-solve structural rewriting defenses like MergeBarrier, yielding a significantly lower recovery accuracy compared to pure parameter-space defenses. This indicates that "structural rewriting" could be a much harder defense direction than "linear obfuscation," although this pathway was not thoroughly explored by the authors.
• ARF applies repulsion only to the attention mechanism because discrete MLP extraction relies on scale-invariant cosine matching, where Euclidean amplification is ineffective. This implies that ARF does not offer symmetric protection against "MLP-side attacks," and its defense boundaries remain to be tested if an attacker specifically targets the MLP.
• The threat model assumes that the attacker has access to "the same pre-trained backbone." If defenders hide or replace the pre-trained anchor during release, the "static anchor" premise of AGA would be shaken, a scenario that is not discussed.
• The conclusion itself acknowledges: this work proves that "linear weight transformations offer only an illusion of security," and robust protection requires new training-aware paradigms—ARF is a patch, not the ultimate solution.

Related Work & Insights

• vs Params / Params-D: Params transforms attention via diagonal matrices and shuffles MLPs via permutations, whereas Params-D adds random dropout to increase extraction difficulty. This paper points out that both diagonal and permutation matrices are subclasses of invertible matrices, which are precisely resolved by AGA's closed-form least squares and Hungarian matching, respectively; dropout also fails to block alignment due to the amplitude disparity.
• vs MergeLock: MergeLock locks the attention parameter space using orthogonal/invertible matrices. Orthogonal matrices fall exactly under Theorem 1, where the error is strictly upper-bounded by \(\|\tau\|_F\), leading to an almost lossless breach.
• vs MergeBarrier: It goes beyond multiplying weights by rewriting MLP topology with Taylor expansions, making inversion ill-posed. This is the only defense that prevents AGA from achieving full recovery; the attack can only bypass the modified blocks and revert to the pre-trained MLP, resulting in a corresponding decrease in recovery accuracy—highlighting that "structural rewriting" is a harder challenge than "linear obfuscation."
• vs MergeGuard: As tuning-stage defenses, MergeGuard uses global L2 regularization to disperse weights and create task conflicts. However, its global constraints degrade single-model performance and can still be breached by AGA. ARF, by contrast, restricts repulsion to the attention mechanisms, which effectively counters the attack while retaining single-model utility, serving as a direct improvement over MergeGuard.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Systematically debunking an entire class of linear parameter defenses using a simple norm observation, backed by dual theoretical theorems, offers a highly incisive perspective.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Full coverage across vision/NLP/generative modalities, three merging paradigms, four defenses, and their compound configurations, complete with hyperparameter sensitivity and single-model fidelity analysis.
• Writing Quality: ⭐⭐⭐⭐ Clear logical flow and balanced presentation of attack and defense; the equations are somewhat dense, though the overall structure is highly readable.
• Value: ⭐⭐⭐⭐⭐ Directly challenges the security foundation of "making models unmergeable" for IP protection, providing clear warnings and guidelines for future research in model copyright defense.

Related Papers

• [ICML 2026] FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language Models
• [AAAI 2026] From Parameter to Representation: A Closed-Form Approach for Controllable Model Merging
• [NeurIPS 2025] Weight Weaving: Parameter Pooling for Data-Free Model Merging
• [ACL 2026] LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging
• [ICCV 2025] FREE-Merging: Fourier Transform for Efficient Model Merging