TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings¶

Conference: ECCV 2026
Paper: ECCV Paper
Area: Information Retrieval
Keywords: Universal multimodal embeddings, task conflict, mixture of experts, low-rank adaptation, hard negatives

TL;DR¶

TSEmbed distributes conflicting multimodal objectives across input-dependent MoE-LoRA experts, then weights hard negatives by routing similarity after expert warm-up, reaching overall scores of 70.5/74.7 with MMEB-only 2B/7B models and surpassing the corresponding B3 baselines by 2.4/2.7 percentage points.

Background & Motivation¶

Universal multimodal embeddings map images, text, and their combinations into a shared vector space so that classification, visual question answering, retrieval, and visual grounding can all be performed through similarity matching. Methods such as VLM2VEC turn multimodal large language models (MLLMs) into encoders, benefiting from their cross-modal interactions and knowledge. However, training these tasks together does not mean that they require identical parameter updates. In Figure 1, Qwen2-VL-7B drops from 70.9 on visual question answering under task-specific training to 57.8 under joint training, a difference of 13.1 percentage points.

The authors diagnose this issue from three perspectives. Task-specific LoRA parameter trajectories separate under PCA projection, suggesting different preferred update directions. Grounding and question answering converge earlier than retrieval and classification, making one stopping point unsuitable for every task. Data-rich retrieval tasks also dominate the jointly trained adapter. These trajectory projections and parameter similarities provide empirical evidence of conflict, not a mathematical proof that all task optima must occupy disjoint spaces. The practical goal is to preserve a universal encoding interface without forcing every objective to compete for one low-rank update.

The paper therefore first changes how adaptation parameters are shared, then uses an internal signal from that structure to improve contrastive learning, rather than simply adding data or an external hard-negative teacher. Core Idea: let a router combine different LoRA experts for different inputs, then, once specialization stabilizes, assign greater contrastive weight to negatives with similar expert-routing paths, addressing both parameter adaptation and embedding boundaries.

Method¶

Overall Architecture¶

The model uses Qwen2-VL as its backbone, accepts multimodal queries or candidate targets, and extracts the final-layer hidden state at the last EOS token as the embedding. Queries, positive targets, and negative targets are all encoded and compared by vector similarity during training. At inference time, the model still produces vectors for retrieval and matching; answer generation or chain-of-thought is not required.

Ordinary LoRA applies the same low-rank residual to every input. TSEmbed replaces it with an input-dependent weighted combination of expert residuals. Training additionally records routing distributions across layers for expert-aware negative weighting, while a two-stage schedule determines when this signal becomes active. Dashed edges below indicate training signals or updates; solid edges indicate encoding or weight computation. The negative-weighting branch is unnecessary at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Queries and<br/>candidate targets"] --> Experts["MoE-LoRA<br/>Conditional Adaptation"]
    Experts --> Embedding["EOS embeddings<br/>Inference similarity matching"]
    Experts -.->|Cross-layer routing distributions| EANS["Expert-Aware<br/>Negative Weighting"]
    EANS -->|Used only in stage two| Schedule["Two-Stage Learning"]
    Embedding -.->|Training similarities| Schedule
    Schedule -.->|Contrastive updates| Experts

Key Designs¶

1. MoE-LoRA Conditional Adaptation: replacing a single shared residual

Alongside each frozen pretrained linear transformation, each expert owns a pair of low-rank matrices producing its own residual output. A linear router reads the current input, applies a temperature-controlled softmax to obtain expert weights, and adds the weighted sum of expert residuals to the backbone output. Two tasks can thus share the pretrained model while emphasizing different experts. An input can also combine multiple experts instead of being assigned to a mutually exclusive task branch. The paper describes softmax weighting, not a top-k sparse activation policy, so it should not be interpreted as executing only one expert per input.

This addresses a limitation of single-adapter LoRA: the issue is not merely total parameter count, but that all samples must use the same update direction. Multiple low-rank subspaces provide different adaptation directions, and the router learns how to combine them. The default uses 4 experts with rank 16 per expert, without explicitly assigning one expert each to classification, question answering, retrieval, or grounding. The authors interpret the favorable performance of 4 experts as matching MMEB's four task categories. This is an empirical interpretation, not proof of a one-to-one expert-to-task correspondence or complete elimination of gradient interference.

2. Expert-Aware Negative Weighting: using internal paths to prioritize negatives

Expert-Aware Negative Sampling (EANS) assumes that a negative target activating similar expert combinations to the query is likely to inhabit a related task-semantic region. Such negatives may be more informative than obviously unrelated cross-task examples. The method collects routing probabilities from all \(L\) layers and \(G\) adapted projection matrices per layer. Each probability vector contains \(N\) experts, yielding a flattened routing signature of length \(L\cdot G\cdot N\). It then measures the L1 distance between query and negative signatures, divided by the total dimensionality. This compares how the network processes its inputs rather than asking a separate teacher to score negatives.

Smaller distances produce raw negative weights closer to the upper bound, while larger distances cause exponential decay toward the lower bound. The default bounds are 10.0 and 0.1, with decay scale \(\sigma=0.002\). Despite the word sampling in its name, the central operation described in the paper is reweighting existing negatives, not retrieving additional examples from a new corpus. Routing similarity is also only a proxy for difficulty: it neither guarantees fine-grained content similarity nor automatically excludes false negatives that should be treated as positives.

To preserve the total weight assigned to negatives, the authors normalize the raw weights over \(M\) negative samples:

\[ {}\tilde{w}_i=w_i\cdot\frac{M}{\sum_{j=1}^{M}w_j}. \]

This is Equation (8) in the paper. The normalized weights sum to the number of negatives and have a mean of 1. Each negative's exponentiated similarity term in the InfoNCE denominator is then multiplied by its weight, while the positive term remains unchanged. Preserving the weight sum is different from preserving the loss denominator: because negative similarities differ, the former does not strictly imply the latter or an identical overall gradient magnitude. The paper's claim of strictly invariant negative contributions therefore requires caution.

3. Two-Stage Learning: learning useful routing before using it to supervise negatives

Randomly initialized routing distributions do not yet contain stable task semantics. Treating similar routes as a hard-negative signal from the outset could therefore amplify arbitrary noise. The first stage uses ordinary InfoNCE alone, allowing experts and routers to specialize through contrastive supervision. After the warm-up threshold, the second stage switches to the EANS-weighted loss. This progressively trains one model; it does not train an external teacher or introduce an additional generative reasoning stage.

The default training run lasts 2,200 steps. EANS is enabled after 600 steps for the 2B model and after 1,200 steps for the 7B model. The longer warm-up for the larger model is an empirical choice in this setting, not a universal scaling ratio. Table 3 directly tests the schedule: the 2B MoE-only model scores 70.20, applying EANS from the outset scores 70.14, and two-stage training reaches 70.52. The substantial benefit of expert-based adaptation must therefore be distinguished from the smaller refinement benefit after routing stabilizes.

A Worked Example¶

Consider an illustrative query consisting of a product image and an attribute request, with the goal of finding a matching product. This is an explanatory example, not a separately reported test case. A training batch contains a matching target, a visually similar target with the wrong attribute, and an obviously unrelated candidate. All inputs pass through the same backbone, but the router separately combines their low-rank residuals before producing comparable EOS embeddings.

During warm-up, ordinary contrastive learning uses embedding similarities alone. After warm-up, the system also compares cross-layer routing signatures. If the product with the wrong attribute happens to follow a route more similar to the query, it receives greater negative weight, emphasizing separation of that confusing pair. The qualification matters: the method does not explicitly identify product attributes, and its routing proxy can fail.

For instance, suppose there are only two negatives with raw weights exactly equal to 10.0 and 0.1. Equation (8) normalizes them to approximately 1.98 and 0.02. These numbers illustrate weight redistribution and are not measured routing outputs. Deployment does not require constructing these two negatives; inputs are encoded and ranked by vector similarity.

Loss & Training¶

The objective throughout training is to bring queries closer to positive targets and separate them from negative targets. Stage one uses standard InfoNCE. Stage two only changes the relative negative weights in its denominator; the described objective does not add a separate task-label classification loss. Equations (2), (3), (4), (6), (7), (9), and (10) contain damaged symbols in the available full-text extraction. Their mechanisms are explained from the surrounding prose here without reconstructing their exact mathematical forms; implementation requires checking the original PDF.

Experiments use 8 NVIDIA A800 GPUs, AdamW, a learning rate of \(5\times10^{-5}\), linear decay, and a global batch size of 1,024, or 128 per GPU. Gradient Caching uses a sub-batch chunk size of 2 to reduce the memory requirements of large-batch contrastive learning. The minimum image pixel count is 401,408, and the LoRA scaling factor is 64.

Routing and contrastive temperatures are distinct from the EANS parameter \(\sigma\): they govern expert-distribution sharpness or similarity logits, whereas \(\sigma\) controls the decay from routing distance to negative weight. The available main text does not fully specify temperature values, the complete list of adapted matrices, how token-level routing becomes a sample signature, or whether gradients are stopped through EANS weights. These details should not be silently assumed when reproducing the method.

Key Experimental Results¶

Main Results¶

Table 1 evaluates 36 MMEB datasets: 10 classification, 10 VQA, 12 retrieval, and 4 grounding datasets. They are also divided into 20 in-distribution (IND) and 16 out-of-distribution (OOD) datasets. The following excerpt compares models in the same Qwen2-VL backbone family trained only on MMEB. Scores follow the paper's hit@1 percentage convention; Overall is not a simple average of the four task-category scores.

Model Scale	Method	Classification	VQA	Retrieval	Grounding	Overall
2B	VLM2VEC	59.0	49.4	65.4	73.4	59.3
2B	B3	67.0	61.2	70.9	79.9	68.1
2B	TSEmbed	68.8	64.3	72.1	85.7	70.5
7B	VLM2VEC	62.6	57.8	69.9	81.7	65.8
7B	B3	70.0	66.5	74.1	84.6	72.0
7B	TSEmbed	71.1	70.3	75.9	91.3	74.7

The actual TSEmbed parameter counts in Table 1 are 2.26B/8.40B; "2B/7B" names the backbone families. The 7B model's IND/OOD scores are 78.8/69.6, exceeding B3's 75.9/67.1 by 2.9/2.5 percentage points. External-data models are not resource-matched competitors: the larger Qwen3-VL-Embedding model, for example, scores 80.1 Overall, above TSEmbed's 74.7. The result should therefore not be generalized into unconditional superiority over all models.

Ablation Study¶

The following excerpt from Table 3 preserves two decimal places. "MoE + EANS, no warm-up" applies EANS without first stabilizing routing; it does not remove MoE.

Model Scale	Config	Overall	Difference from Same-Scale MoE-only
2B	VLM2VEC	59.30	-10.90
2B	MoE-only	70.20	0.00
2B	MoE + EANS, no warm-up	70.14	-0.06
2B	Full model	70.52	+0.32
7B	VLM2VEC	65.80	-8.41
7B	MoE-only	74.21	0.00
7B	MoE + EANS, no warm-up	74.18	-0.03
7B	Full model	74.68	+0.47

Key Findings¶

MoE-LoRA accounts for most of the improvement: it gains 10.90/8.41 percentage points over VLM2VEC at 2B/7B. EANS with warm-up adds 0.32/0.47 percentage points. Attributing the entire improvement to hard-negative weighting would be misleading.
Table 2 reports zero-shot results on proprietary production datasets: advertising Recall rises from 11.33% to 33.20%, a gain of 21.87 percentage points, not a relative increase of 21.87%; theme NDCG@5 rises from 84.17% to 86.22%. These are not online A/B revenue or click-through-rate gains.
Figure 7 evaluates 2, 4, 6, and 8 experts. Four experts provide more robust cross-task performance, while eight generally degrade performance. The paper does not establish that more experts or larger task collections necessarily yield better results.
The discussion of Figure 5 reports fluctuations of approximately 1 percentage point over \(\sigma\in[2\times10^{-5},2\times10^{-1}]\). This retains the authors' approximate summary without inventing exact curve values from the damaged figure extraction.

Highlights & Insights¶

Task conflict and hard-negative learning are connected through routing: it participates in representation computation and exposes internal structure among training examples. Reusing this signal avoids a separate negative-scoring teacher, although routing-signature aggregation and distance calculations still incur costs.
The ablation makes training order meaningful: EANS behaves differently when attached to random routing versus mature routing. A transferable lesson is to establish that an internal signal carries useful semantics before using it for self-supervised weighting.
A universal interface does not require fully shared adaptation parameters. One vector encoder can retain unified inputs and outputs while supporting multiple internal adaptation directions, a useful alternative to separate full backbones for every retrieval domain.

Limitations & Future Work¶

Routing distance is a semantic proxy, not a relevance label. Truly relevant targets from nearby tasks may be emphasized as false negatives. Future work could filter false negatives and directly evaluate routing distance against human relevance judgments and negative difficulty.
Parameter overhead alone does not establish low deployment cost. Table 4 reports A100 inference time on 5,630 advertising examples rising from 35.72 to 44.67 minutes for 2B, and on 6,532 gaming examples from 49.58 to 70.02 minutes, approximately 25.1% and 41.2% increases. The paper's description of latency as "negligible" does not reflect these relative increases.
Evidence for task scaling primarily consists of architectural replacement and expert-count sweeps on a fixed MMEB task collection. The paper does not show scaling curves with increasing task counts or forgetting after adding new tasks. The relationship between expert counts and changing task-cluster structure remains to be tested.
There is no dedicated limitations section or multi-seed error bars. EANS improves only modestly over MoE-only, so repeated runs are needed to establish stability. Proprietary production data and incompletely specified implementation details also limit independent verification.
Table 1 lists 7B VLM2VEC IND/OOD scores of 65.2/56.3 but an Overall score of 65.8, higher than both. This suggests a reporting or formatting issue. This note preserves the reported Overall score without correcting it and does not derive conclusions from that baseline's distribution-specific columns.

vs VLM2VEC: Both use MLLMs as contrastively trained embedding encoders. TSEmbed changes low-rank parameter sharing and redistributes negative weights after routing stabilizes; it does not improve retrieval by increasing generation length.
vs B3 / QQMM-embed: B3 focuses on batch construction, while QQMM-embed amplifies hard-negative gradients. TSEmbed additionally addresses parameter-level interference and uses expert utilization to determine negative weights. The paper does not test all these strategies together, so their gains cannot be assumed additive.
vs PCGrad / GradNorm: These methods directly manipulate gradients or task weights, whereas TSEmbed indirectly reduces sharing conflicts through conditional adaptation. They are not directly compared in the main result table, so claims of greater efficiency are not established by a controlled comparison here.
Further research: Separately fix total adaptation parameters and expert count while progressively adding task clusters to determine whether gains arise from extra capacity, conditional routing, or alignment with task structure. These are questions motivated by the evidence boundaries, not experiments already completed in the paper.

Rating¶

Novelty: 4/5. MoE-LoRA is not itself a new component, but combining cross-layer routing-based negative weighting with warm-up has a clear methodological rationale.
Experimental Thoroughness: 4/5. Two model scales, public and proprietary tasks, and multiple ablations are covered, but multi-seed statistics and actual task-count scaling experiments are missing.
Writing Quality: 3/5. The method is coherent, but claims about weight normalization, efficiency, and some tabulated values need tighter qualification.
Value: 4/5. Conditional adaptation is a reusable approach for multitask retrieval encoders, subject to latency and reproducibility costs.