Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/2430.pdf
Area: Multimodal VLM
Keywords: embedding backward compatibility, multimodal retrieval, adapter, backward compatible training, multimodal large language models
TL;DR¶
To eliminate the prohibitive computational cost of database re-indexing during multimodal retrieval system upgrades, this paper presents MKP-Adapter—the first adapter-only backward compatible training framework for Multimodal Large Language Models (MLLMs)—which enforces compatibility and retains representation quality via multi-level geometric preservation losses and focal re-weighting without touching backbone weights.
Background & Motivation¶
In large-scale multimodal information retrieval systems, dual-tower encoders (such as CLIP) and multimodal large language models (such as E5-V, GME, and UniME) function as core embedding generators that map text, images, visual documents, and video frames into dense representation spaces for semantic search. However, when a system is upgraded with a more performant embedding model trained independently, the semantic distributions between the old and new latent spaces drift substantially, rendering their embeddings mutually incompatible. As a result, deployment typically mandates rebuilding the entire gallery database—a re-indexing process known as backfilling—which incurs massive computational overhead, service disruption, and long deployment latency when scaling to millions or billions of items.
To avert backfilling, Backward Compatible Training (BCT) incorporates compatibility constraints during model optimization so that new query embeddings can directly search old database galleries. Nonetheless, conventional BCT paradigms require retraining the new model from scratch or performing parameter-efficient tuning (e.g., LoRA tuning on the backbone as in XBT). Updating the backbone model presents three fundamental bottlenecks in the era of MLLMs: first, repeatedly passing inputs through billion-parameter foundation models during training is computationally exorbitant; second, imposing compatibility constraints directly on the backbone risks distorting pre-trained multimodal representations and causing catastrophic performance regression; third, many cutting-edge models are exposed solely via black-box commercial APIs where internal model weights are inaccessible.
A practical alternative is to learn a lightweight post-projection adapter while keeping the new backbone completely frozen. However, naive regression toward the old embeddings forces the adapter to fit into a comparatively weaker, noisier semantic manifold, thereby degrading the fine-grained discriminative power of the new embeddings and compromising backward compatibility. Core idea: this paper introduces MKP-Adapter, a backbone-free adapter-only BCT approach that reframes backward compatibility from naive target fitting to active representation preservation, combining point-wise cycle reconstruction, pairwise distance preservation, triplet angle alignment, and focal re-weighting to maintain the geometric manifold of the new embeddings during projection.
Method¶
Overall Architecture¶
MKP-Adapter acts as a compact multi-layer perceptron projection mapping \(h: \mathbb{R}^{d_{\text{new}}} \to \mathbb{R}^{d_{\text{old}}}\). Throughout training and inference, both the old embedding model \(f_{\text{old}}\) and the new model \(f_{\text{new}}\) remain strictly frozen. The training procedure operates entirely on pre-extracted, L2-normalized embeddings, eliminating any forward or backward passes through foundation model backbones.
The optimization objective balances two complementary goals: a base projection loss minimizes the Euclidean distance between projected features \(h(\mathbf{x}^{\text{new}})\) and target old embeddings \(\mathbf{x}^{\text{old}}\), while a self-supervised multi-level preservation regularizer enforces geometric fidelity across individual samples, pairwise distances, and triplet angles, dynamically guided by focal re-weighting for difficult instances.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Pre-extracted New Embeddings<br/>x_new"] --> B["Base Adapter Mapping h<br/>project to old space h(x_new)"]
B --> C["Point-wise Cycle Reconstruction<br/>auxiliary inverse mapping g reconstructs x_new"]
B --> D["Multi-level Geometric Preservation<br/>pairwise distances and triplet angles aligned"]
C --> E["Focal Re-weighting Strategy<br/>adaptive scaling based on reconstruction error"]
D --> E
B --> F["Base Projection Loss<br/>minimize distance to target x_old"]
E --> G["Combined Objective Optimization<br/>L_total = L_base + lambda * L_focal"]
F --> G
G --> H["Inference Deployment<br/>frozen backbone + adapter h directly queries old gallery"]
Key Designs¶
1. Point-wise cycle reconstruction: preserving individual semantic fidelity via inverse mapping
When high-dimensional representations from an expressive new model are projected into an older embedding space, single-sample semantic details are susceptible to compression collapse. To guarantee that each projected vector \(h(\mathbf{x}_i^{\text{new}})\) retains the essential semantic attributes of the original sample, the method introduces a cycle-consistency reconstruction mechanism. An auxiliary adapter \(g: \mathbb{R}^{d_{\text{old}}} \to \mathbb{R}^{d_{\text{new}}}\) is trained to invert the projected embedding back into the new representation space:
Minimizing this reconstruction error guarantees that \(h\) acts as an information-preserving transformation capable of recovering the initial representation. Crucially, the auxiliary adapter \(g\) is utilized exclusively during training and discarded at deployment, thereby introducing zero latency or memory footprint during live query execution.
2. Multi-level geometric preservation: anchoring second- and third-order relational manifolds
Because retrieval performance depends fundamentally on relative ranking rather than isolated vector coordinates, preserving individual embeddings alone cannot guarantee retrieval consistency. The method establishes second- and third-order relational preservation losses using the new embeddings themselves as self-supervised targets, dispensing with external teacher models. For pairwise relationships, a distance-wise preservation loss aligns relative distances across both randomly sampled mini-batch pairs (global) and pre-computed \(m\)-nearest neighbors (\(m=100\)) in the new embedding space (local):
where \(\text{Dist}(\mathbf{a}, \mathbf{b}) = 1 - \cos(\mathbf{a}, \mathbf{b})\). To prevent manifold distortion and angular rotation during dimensionality adaptation, the framework further introduces an angle-wise preservation loss on sample triplets:
where \(\psi_A(\mathbf{x}_i, \mathbf{x}_j, \mathbf{x}_k) = \langle \frac{\mathbf{x}_i - \mathbf{x}_j}{\|\mathbf{x}_i - \mathbf{x}_j\|_2}, \frac{\mathbf{x}_k - \mathbf{x}_j}{\|\mathbf{x}_k - \mathbf{x}_j\|_2} \rangle\) represents the cosine of the angle centered at vertex \(\mathbf{x}_j\). This joint second- and third-order regularization preserves the intrinsic metric geometry of the new model across projection.
3. Focal re-weighting strategy: prioritizing challenging multimodal samples
Multimodal retrieval datasets feature diverse inputs—ranging from simple object captions to complex infographics and multitrack videos—which differ markedly in alignment difficulty. Under standard uniform averaging, easily projected pairs dominate gradient updates, leaving difficult semantic transitions under-optimized. Drawing inspiration from Focal Loss in object detection, the approach dynamically scales loss terms based on individual reconstruction residuals. For sample \(i\), the reconstruction error \(\Delta_i = \| g(h(\mathbf{x}_i^{\text{new}})) - \mathbf{x}_i^{\text{new}} \|_2^2\) is normalized against the batch mean \(\bar{\Delta} = \frac{1}{N} \sum_{i=1}^N \Delta_i\):
where \(\gamma\) is a focusing parameter (set to \(1.0\) by default). Analogous focal weightings are applied to the distance-wise and angle-wise objectives. By adaptively allocating higher optimization priority to distorted representations, this mechanism substantially enhances robustness on out-of-domain evaluation benchmarks.
Loss & Training¶
The overall training objective combines the base projection loss with the focally re-weighted multi-level preservation regularizer:
where \(\mathcal{L}_{\text{base}} = \sum_{i=1}^N \| h(\mathbf{x}_i^{\text{new}}) - \mathbf{x}_i^{\text{old}} \|_2^2\) and the balancing weight \(\lambda\) is set to \(1.0\). The adapter adopts a 2-layer MLP architecture (LayerNorm \(\to\) Linear \(\to\) GELU \(\to\) Dropout(0.1) \(\to\) Linear) with hidden dimensionality matching the input feature dimension. Optimization is conducted via AdamW with an initial learning rate of \(10^{-3}\) and a cosine annealing schedule over 50 epochs (or as few as 10 epochs). Because all representations are pre-extracted, training converges within 1–2 hours on 8 A100 GPUs.
Key Experimental Results¶
Main Results¶
The method is systematically evaluated across two upgrade paradigms: upgrading from dual-tower architectures to MLLMs (CLIP ViT-L/14 \(\to\) GME-7B / UniME-7B) and upgrading between MLLM generations (E5-V-7B \(\to\) UniME-7B). Results on standard cross-modal image-to-text (I2T) and text-to-image (T2I) benchmarks are reported via Recall@1 (%).
| Upgrade Scenario | Method | MS-COCO (I2T / T2I) | Flickr30K (I2T / T2I) | Urban1K (I2T / T2I) | Average (I2T / T2I) |
|---|---|---|---|---|---|
| CLIP \(\to\) GME-7B (Old Model) | x_old (Upper/Old Ref) | 56.32 / 36.50 | 85.10 / 65.00 | 68.00 / 55.90 | 69.80 / 52.47 |
| CLIP \(\to\) GME-7B (New Target) | x_new (Target Upper) | 68.68 / 57.35 | 90.80 / 80.90 | 90.10 / 88.60 | 83.20 / 75.61 |
| CLIP \(\to\) GME-7B (Compatibility) | Base adapter | 58.38 / 34.93 | 85.50 / 64.74 | 62.00 / 51.00 | 68.63 / 50.22 |
| CLIP \(\to\) GME-7B (Compatibility) | EC-style adapter | 59.52 / 35.88 | 85.30 / 65.74 | 63.50 / 50.60 | 69.44 / 50.74 |
| CLIP \(\to\) GME-7B (Compatibility) | CL-based adapter | 54.38 / 37.06 | 76.00 / 68.00 | 63.40 / 64.10 | 64.59 / 56.38 |
| CLIP \(\to\) GME-7B (Compatibility) | MKP-Adapter (Ours) | 66.18 / 38.78 | 87.50 / 69.16 | 68.70 / 77.60 | 74.13 / 61.85 |
| CLIP \(\to\) UniME-7B (Compatibility) | Base adapter | 53.74 / 37.44 | 81.90 / 66.22 | 54.50 / 65.50 | 63.35 / 56.39 |
| CLIP \(\to\) UniME-7B (Compatibility) | EC-style adapter | 56.22 / 38.50 | 82.20 / 67.70 | 58.40 / 64.80 | 65.61 / 57.00 |
| CLIP \(\to\) UniME-7B (Compatibility) | CL-based adapter | 53.00 / 38.42 | 79.40 / 68.76 | 69.30 / 70.70 | 67.23 / 59.29 |
| CLIP \(\to\) UniME-7B (Compatibility) | MKP-Adapter (Ours) | 66.68 / 42.22 | 91.20 / 72.00 | 75.40 / 75.00 | 77.76 / 63.07 |
| E5-V \(\to\) UniME-7B (Compatibility) | Base adapter | 55.90 / 50.21 | 85.10 / 79.22 | 77.80 / 86.90 | 72.93 / 72.11 |
| E5-V \(\to\) UniME-7B (Compatibility) | EC-style adapter | 57.30 / 50.32 | 85.90 / 79.52 | 78.10 / 88.10 | 73.77 / 72.65 |
| E5-V \(\to\) UniME-7B (Compatibility) | CL-based adapter | 63.18 / 50.12 | 88.70 / 78.46 | 81.70 / 88.70 | 77.86 / 72.43 |
| E5-V \(\to\) UniME-7B (Compatibility) | MKP-Adapter (Ours) | 65.28 / 52.61 | 90.60 / 81.10 | 90.20 / 92.10 | 82.03 / 75.27 |
Across the 36 sub-datasets of MMEB, MKP-Adapter delivers an overall Precision@1 of 49.56% in the CLIP \(\to\) UniME setting, substantially surpassing the old model (40.29%) and outperforming Base (20.03%), EC (20.57%), and CL-based (39.27%) baselines. On ViDoRe-V1 visual document retrieval, MKP-Adapter achieves 45.35% nDCG@5 (+17.05% over the old model at 28.30%). On MSR-VTT, MSVD, and VATEX video retrieval, it attains an average Recall@1 of 36.25% (+7.85% over the old baseline at 28.40%).
Ablation Study¶
A component-wise ablation under the CLIP ViT-L/14 \(\to\) UniME-7B scenario illustrates the contribution of each module:
| Configuration | I2T Average R@1 | T2I Average R@1 | MMEB Average P@1 | Note |
|---|---|---|---|---|
| Full MKP-Adapter | 77.76 | 63.07 | 49.56 | Complete multi-level preservation + focal weighting |
| w/o Point-wise preservation | 77.11 | 63.22 | 47.43 | Loss of individual fidelity; MMEB drops by 2.13% |
| w/o Distance-wise preservation | 71.74 | 58.08 | 47.81 | Metric ranking disrupted; I2T falls 6.02%, T2I falls 4.99% |
| w/o Angle-wise preservation | 76.83 | 62.26 | 41.55 | Manifold geometry degraded; MMEB plunges by 8.01% |
| w/o Focal re-weighting | 77.43 | 62.98 | 46.61 | Insufficient gradient on hard samples; MMEB drops 2.95% |
Key Findings¶
- Distance preservation governs cross-modal ranking performance: Disabling pairwise distance preservation incurs a steep performance cliff on standard image-text retrieval (I2T drops by 6.02% and T2I drops by 4.99%), underscoring that preserving relative metric distance is vital for ranking discriminability.
- Angle preservation secures instruction and out-of-domain generalization: Omitting the third-order angle constraint causes MMEB Precision@1 to plummet from 49.56% to 41.55% (an 8.01% decline), demonstrating that triplet angular alignment preserves global geometric structure against domain shift.
- Minimal representation degradation in self-compatibility testing: Evaluating retrieval when both query and database employ projected embeddings \(h(\mathbf{x}^{\text{new}})\) yields 86.30% I2T and 77.11% T2I (E5-V \(\to\) UniME), trailing the unprojected new model (86.52% / 77.15%) by less than 0.22 points. This indicates that MKP-Adapter preserves virtually all native representation power.
- Negligible computational and latency overhead: The 2-layer MLP introduces only 0.25 ms latency per batch of 32 on a single A100 GPU (0.01% of the backbone forward pass) and trains in 1–2 hours on 8 A100 GPUs using 558K pre-extracted embeddings.
Highlights & Insights¶
- Shifting from teacher supervision to self-supervised manifold preservation: While conventional knowledge distillation and forward-compatible training rely on a superior teacher model, adapter-only backward compatibility faces a weaker, noisier target space. MKP-Adapter overcomes this by using the new embeddings themselves as self-supervised anchors to safeguard manifold geometry.
- Plug-and-play decoupling for proprietary API backbones: By operating solely on pre-extracted embeddings without requiring gradient access to model backbones, the method provides an off-the-shelf backward compatibility pathway for proprietary black-box APIs and high-resolution document/video retrieval systems.
Limitations & Future Work¶
- Dependence on overlap between representation capacities: When the old embedding model exhibits very low dimensionality or severe capacity bottlenecks, a lightweight MLP projection cannot bridge the representational gap without semantic loss, setting an upper bound bounded by the intrinsic rank of the old embedding space.
- Static offline embedding assumption: Training relies on an offline pre-extracted pool of multimodal embeddings. If the downstream deployment shifts to previously unseen multimodal domains or novel formats, the frozen adapter cannot dynamically adjust backbone attention maps. Future work could investigate test-time adaptation strategies without backbone tuning.
Related Work & Insights¶
- vs Conventional Backward Compatible Training (BCT / Basis Transformation): Prior BCT frameworks enforce compatibility during pre-training or fine-tuning of the new model, preventing decoupled model development. MKP-Adapter acts strictly post-hoc as a lightweight modular adapter.
- vs XBT (Cross-Modal Backward-Compatible Learning): XBT still requires two-stage LoRA tuning of the vision-language backbone, introducing training risks and requiring weight access. MKP-Adapter keeps backbones frozen, consumes orders of magnitude less compute, and achieves superior compatibility on MLLMs.
- vs Embedding Converter (EC): EC targets forward-compatible training for text retrieval using pairwise distance distillation. MKP-Adapter addresses the harder backward-compatible multimodal regime via joint cycle reconstruction, triplet angular preservation, and focal re-weighting.
Rating¶
- Novelty: 4.5/5 Formulates the first adapter-only backward compatibility framework for MLLMs with elegant self-supervised geometric preservation.
- Experimental Thoroughness: 5.0/5 Validated across image-text, MMEB 36-task benchmark, ViDoRe visual document retrieval, and three video retrieval datasets with granular ablations.
- Writing Quality: 4.5/5 Thorough problem formulation, clear mathematical rigor, and transparent experimental narratives.
- Value: 5.0/5 Addresses a mission-critical engineering hurdle in production search and recommendation architectures, removing the need for costly database re-indexing.