Training-Free Task Classification for Multi-Task Model Merging¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/BAIKLAB/SiM
Area: Model Compression
Keywords: dynamic model merging, training-free routing, task classification, projection residuals, compressed task vectors

TL;DR¶

SiM identifies a test input's task using low-rank feature subspaces built from small task support sets, then selects a compressed task vector, reaching 92.7% average accuracy on eight tasks with CLIP ViT-B/32 versus 92.8% for individual experts, without router training or test-time task IDs.

Background & Motivation¶

Fine-tuning the same pretrained model on scene, vehicle, and digit datasets produces multiple experts of similar parameter size. Keeping every full expert at deployment makes storage grow with the number of tasks; collapsing their parameters into a static model risks interference between task updates. Task Arithmetic adds task vectors, TIES-Merging addresses parameter sign conflicts, and TSV-M modifies the singular-vector structure of task updates, but these static approaches still make every input share one fixed parameter configuration. Consequently, strong individual experts do not guarantee that their merged model retains each expert's capabilities.

Dynamic model merging lets the input determine which task parameters to use, but creates another deployment question: what identifies the appropriate expert? Methods such as TWIN-Merging learn a router, requiring extra training data and optimization; DaWin avoids router training but uses multiple models to compute input-dependent signals. Compact task representations such as EMR and TALL-Mask reduce storage, yet typically rely on knowing the task identity when applying the corresponding update. Storing task knowledge compactly and invoking it correctly without task identity are separate problems; parameter compression does not automatically solve the latter.

The paper exploits dataset-specific signals already present in pretrained features: images or texts from different sources often occupy different low-dimensional regions. This does not assume that the model understands arbitrary task instructions; a small set of inputs from each registered task defines a geometric signature instead. Comparing test inputs against these signatures supplies existing compressed task representations with a selector that needs no gradient-based training. Core Idea: replace a trainable router with task-subspace projection residuals, decoupling task identification from compressed-expert recovery so that existing expertise can be invoked without a supplied task ID.

Method¶

Overall Architecture¶

SiM stands for Singular-vector-based Manifold; it takes a test input and produces a downstream class prediction or task answer. Offline preparation requires a shared pretrained backbone, existing fine-tuned task experts, and small support sets organized by task; expert training is outside SiM's training-free claim. The stages are Task Subspace Construction, Projection Residual Routing, and Compressed Updates and Grouped Inference: store task signatures, predict the task ID, then recover the relevant parameter update and execute the downstream model. The first stage computes statistics from support inputs, while the latter stages process actual test inputs without revisiting the original support sets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Support["Per-task support inputs"] --> Build["Task Subspace<br/>Construction"]
    Build -->|"Store means and bases offline"| Route["Projection Residual<br/>Routing"]
    Input["Pretrained features of test inputs"] --> Route
    Route -->|"Predicted task ID"| Execute["Compressed Updates<br/>and Grouped Inference"]
    Updates["Existing compressed task vectors"] --> Execute
    Execute --> Output["Downstream predictions"]

Here, merging does not mean that a single immutable weight configuration serves the entire inference process. The system retains shared base parameters and compact task updates, with routing selecting the update currently in use; it remains an input-adaptive dynamic model. Nor must each test input pass through every task expert: task selection mainly occurs in one shared pretrained feature space.

Key Designs¶

1. Task Subspace Construction: store distribution directions, not just a center

For each task, the authors randomly sample support inputs from its training set, extract features with the frozen pretrained model, and calculate their mean. Subtracting this mean from each feature produces a centered matrix whose dimensions are support-set size by feature dimension, to which singular value decomposition is applied. Keeping the leading right singular vectors gives a low-rank linear approximation to the task's variations around its mean. Each task ultimately stores a mean and a subspace basis, rather than all support samples or a newly trained task-discrimination network. The mean describes location, while the basis describes variations still compatible with the task; together they define an affine approximation. The word manifold refers to this low-rank approximation, not an additional learned nonlinear manifold model.

Figure 4 provides empirical motivation: across eight vision tasks, the leading 10% of singular-value directions capture more than 80% of the Frobenius energy. Most feature variation therefore lies along a few directions, and distances that treat all directions equally discard useful structure. The main text defaults to \(N=32\) support samples per task and sets \(k\) to 10% of feature dimension \(d\). However, centering bounds the sample matrix rank by \(N-1\), so the number of identifiable nonzero directions remains limited by support-set size. This is a linear-algebra constraint relevant to reproduction; the main text does not explain how nominal \(k\) is clipped when it exceeds effective rank, so that implementation detail remains unverified. Support inputs require task membership, but this construction does not optimize a classification loss using their downstream class labels.

2. Projection Residual Routing: compare the feature component a task subspace cannot explain

At test time, the same pretrained backbone extracts an input feature, from which each task mean is separately subtracted. For each centered vector, the method removes its projection along the task subspace, retaining the orthogonal component and using its magnitude as a mismatch score. An input far from the mean can still have a small residual if its displacement follows directions common to that task. Conversely, an input at the same Euclidean distance has a larger residual if its displacement follows directions outside the task subspace. SiM selects the task with the smallest residual: this is deterministic hard routing, not a learned task probability or a weighted ensemble of expert outputs. The mechanism distinguishes distance from a center from deviation from task structure; Figure 5 illustrates this with equal-distance features pointing in different directions.

The following is a mechanism-level expression organized from the prose in Section 4.3; cached Equations (5)–(6) are corrupted in extraction, so this is not presented as a verbatim transcription. Here \(\mathbf z\) is the input feature, \(\boldsymbol\mu_t\) is the task mean, and the columns of \(\mathbf V_{t,k}\) are the retained orthonormal basis vectors.

\[ r_t(\mathbf z)=\left\|(\mathbf I-\mathbf V_{t,k}\mathbf V_{t,k}^{\top})(\mathbf z-\boldsymbol\mu_t)\right\|_2, \qquad \hat t=\operatorname*{arg\,min}_t r_t(\mathbf z). \]

Section 4.4 additionally analyzes the ratio between residual magnitude and centered-feature length to remove scale differences when examining directional alignment. This residual ratio is an analysis quantity used in Figure 6 and should not be confused with the original projection residual used for routing in the main text. The diagonal entries of the average ratio matrix in Figure 6 are the smallest, supporting the empirical observation that each task subspace better explains its own inputs. Average separation does not make every individual input separable, however; the task-identification experiments still show confusion, especially for the broader scene task.

3. Compressed Updates and Grouped Inference: use the predicted task to retrieve compact expertise

After predicting the task ID, SiM retrieves the matching task update from an existing compressed representation rather than introducing another parameter-compression rule. EMR constructs a shared merged vector and recovers task updates through task-specific binary masks and rescaling factors; TALL-Mask also uses task masks, with a shared vector obtained from either Task Arithmetic or TIES-Merging. TSV-C instead retains a small number of singular components of each layer's task matrix, representing the update through low-rank factors. These methods supply compact knowledge indexed by task ID, while SiM supplies the missing task-ID prediction. Parameter-space low-rank compression is distinct from the previous design's feature-space subspaces: the former determines what is stored, and the latter determines what is retrieved.

With shared pretrained parameters \(\theta_0\) and compressed task update \(\Delta_t\), the executed model can be summarized as:

\[ {\theta}_{\mathrm{run}}=\theta_0+\Delta_{\hat t}. \]

For a mixed-task batch, the system first predicts every sample's task, groups inputs by predicted ID, and runs each group with the corresponding update. Grouping avoids the poor throughput of constructing different parameters sample by sample, but does not eliminate feature extraction, grouping, or weight-switching costs. Table 1 gives the task-basis storage term as \(O(Tdk)\), plus compressed task parameters; the means and shared backbone must also be stored. Cost therefore still grows with task count \(T\), although storing every full expert simultaneously is no longer necessary. The efficiency of the dynamic system depends jointly on the compression backend and batch scheduling, not solely on a distance formula.

A Worked Example¶

Suppose the system has registered eight vision tasks and receives a EuroSAT image without an externally supplied dataset name. Offline preparation has already stored means and bases for EuroSAT, Cars, DTD, and the other tasks, alongside their compressed updates. Online processing encodes the image with pretrained CLIP and evaluates its projection residual under each of the eight task subspaces. If EuroSAT has the smallest residual, the input joins the EuroSAT group and uses that task update with other inputs assigned to the same task in the batch. The resulting model must still identify the land-cover class; routing identifies the source task rather than completing land-cover classification itself. This is an explanatory example, not a reported per-sample experiment; the 99.9% EuroSAT task-identification accuracy in Table 6 is not its land-cover classification accuracy.

Loss & Training¶

SiM introduces no router-training loss or router back-propagation; offline work consists of feature computation, mean estimation, and SVD. Existing expert fine-tuning, compression-backend construction, and support-data acquisition still carry costs. No data during merging and inference means no further access to the support set, not inference without test inputs or a system that has never used data. Figure 8 indicates that 32 support samples already approach task-identification performance obtained with all training data, but the local cache lacks the appendix's per-setting numbers. The authors also refer rank selection, support-set selection, soft versus hard routing, and intermediate-layer analysis to the appendix; those details lie outside the evidence verified here.

Key Experimental Results¶

Main Results¶

Table A selects results from original Table 2, page 10; the metric is average vision-task classification accuracy (%), showing CLIP ViT-B/32 only. The eight tasks are SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD; the fourteen- and twenty-task settings progressively add datasets. SiM operates without test-time task IDs; Fine-tuned is the individual-expert reference, not a single deployment system with an equivalent storage budget.

Method	8 tasks	14 tasks	20 tasks
Fine-tuned	92.8	90.9	91.3
Task Arithmetic	70.8	65.3	60.5
TSV-M	85.7	80.1	77.1
WEMoE	90.4	83.1	74.4
MoW-Merging	88.1	83.2	79.3
EMR + SiM	90.7	87.0	86.1
TM-TA + SiM	92.3	89.5	89.6
TM-TIES + SiM	92.7	88.5	82.2
TSV-C + SiM	92.1	89.4	88.6

TM-TIES + SiM trails the experts by only 0.1 percentage points on eight tasks, but TM-TA + SiM is stronger on twenty tasks, showing that routing does not remove backend differences. Original Table 3, page 11, also reports seven NLP tasks with T5-large: TM-TIES + SiM averages 85.6%, compared with 83.4% for WEMoE and 88.8% for individual experts. The same configuration reaches only 49.0% on WSC versus 79.2% for the expert; overall improvements should not hide task-level degradation.

Ablation Study¶

Table B selects results from original Table 6, page 13; the metric is task-identification accuracy (%) in the CLIP ViT-B/32 eight-task setting, not downstream classification accuracy. It shows the average and two informative tasks: EuroSAT demonstrates the benefit of structural modeling, while SUN397 shows that this benefit is not universal across tasks.

Task-identification criterion	EuroSAT	SUN397	Eight-task average
Euclidean distance	86.5	97.8	95.8
Cosine similarity	86.5	97.0	95.5
Mahalanobis distance	95.4	99.8	96.5
k-NN (k=1)	99.2	75.7	92.0
SiM projection residual	99.9	95.4	99.0

Projection residuals improve the average over Euclidean distance by 3.2 percentage points and improve EuroSAT by 13.4 percentage points. On SUN397, however, SiM is 2.4 percentage points below Euclidean distance and also below Mahalanobis distance; the accurate conclusion is best on average, not best on every task.

Table C selects results from original Table 5, page 13, using an NVIDIA GeForce RTX 3090, CLIP ViT-B/32, and eight vision tasks. Latency is seconds per input and VRAM is in GB; these results reflect the paper's implementation and batching setup, not a fixed speedup across devices.

Method	Latency per input (s)	VRAM (GB)	Average accuracy (%)
TSV-M	0.0008	1.3	85.7
TWIN-Merging	0.03	3.2	84.0
DaWin	0.63	5.5	89.0
WEMoE	0.02	4.3	90.4
TM-TIES + SiM	0.003	1.8	92.7
TSV-C + SiM	0.003	1.6	92.1

Key Findings¶

Table 2 combines routing with compressed updates, and the best backend changes with task scale; SiM should not be treated as a standalone expert-compression algorithm.
In Table 5, TSV-C + SiM has 10 times lower latency than TWIN-Merging but remains slower than static TSV-M; accuracy, memory, and latency represent distinct trade-offs.
Original Table 4, page 12, reports 92.7% on eight seen tasks, 63.3% on six unseen tasks, and 80.1% overall across fourteen tasks for TM-TIES + SiM.
Unseen inputs are still routed to seen-task subspaces rather than new experts; this supports some transfer ability, not a solution to open-set task identification.

Highlights & Insights¶

Treating task identification geometrically turns existing pretrained features into a routing signal without training another task-classification network.
Feature bases and parameter-compression bases have separate responsibilities and communicate through a predicted task ID, enabling interchangeable compression backends.
Hard routing simplifies selection and enables task-grouped minibatches; the system benefit comes from reusing parameter configurations, not merely cheap residual calculations.

Limitations & Future Work¶

Training-free does not mean support-data-free: task signatures still require accessible inputs with known task membership, and storing statistics does not automatically guarantee privacy.
The method exploits dataset-source signals; overlapping tasks, multiple objectives on one image, or domain shift may weaken separability, and the main text gives no rejection rule for hard routing.
Effective-rank handling with small support sets requires implementation verification; a nominal fraction of feature dimension should not automatically be interpreted as the effective subspace dimension.
The local full text covers the main paper and references but lacks the cited appendix; fifty-task results, fine-grained tasks, and oracle-routing gaps remain author statements rather than independently checked data here.
Reader suggestion: investigate confidence calibration and rejection using existing residuals, and report how routing errors affect each downstream task rather than only average accuracy.

vs Task Arithmetic / TIES-Merging: they mainly determine static merged weights, whereas SiM selects which task update each input invokes; the methods address different levels of the problem.
vs TWIN-Merging / DaWin: the former learns a router and the latter derives dynamic weights from model-prediction signals; SiM makes a hard selection using stored feature-geometric signatures.
vs EMR / TALL-Mask / TSV-C: these methods provide compact task representations, while SiM supplies selection without a known task ID; experiments must be interpreted together with their specific backend.
Research direction: use the gap between the smallest residual and competing task residuals as a possible rejection signal, then test whether it predicts downstream failure; this is a reader-proposed extension, not a validated result of the paper.

Rating¶

Novelty: 4/5. Feature-subspace residuals fill a clear gap in training-free routing for model merging, although the underlying tools are classical.
Experimental Thoroughness: 4/5. Vision, NLP, task scaling, and runtime costs are covered, but the missing appendix leaves some reproduction details unverified.
Writing Quality: 4/5. The argument is clear, but task identification must be distinguished from downstream performance, and corrupted cached equations limit exact formula verification.
Value: 4/5. Directly relevant to deploying existing task experts and compact updates, with realistic mixed tasks and unknown domains still requiring further validation.