Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

Conference: ECCV 2026
arXiv: 2602.05275
Code: https://github.com/xxx (None)
Area: Multimodal VLM
Keywords: Multimodal Embedding, Visual Token Compression, Progressive Training, MLLM-as-a-Judge, Universal Retrieval

TL;DR

Magic-MM-Embedding introduces a parameter-free visual token compression mechanism (reducing tokens to 25%) based on InternVL3, coupled with a three-stage progressive training pipeline (generative recovery \(\rightarrow\) contrastive pre-training \(\rightarrow\) MLLM judge fine-tuning). This approach not only overcomes the performance degradation caused by compression but also completely outperforms prior SOTA models on MMEB, VisDoc, and cross-modal retrieval, while significantly reducing inference latency.

Background & Motivation

Background: Multimodal embedding models are transitioning from the CLIP dual-tower architecture to the MLLM paradigm. MLLMs treat visual features as discrete tokens alongside text tokens in a unified transformer, achieving token-level deep cross-modal fusion. This demonstrates capabilities far exceeding CLIP on universal multimodal retrieval. Recent work continuously pushes performance boundaries in terms of data scale, hard negative mining, and multi-stage progressive training.

Limitations of Prior Work: These methods ignore a critical bottleneck—the massive computational overhead caused by excessively long visual token sequences. A standard MLLM (such as LLaVA-OneVision) can generate up to 7,290 visual tokens for a single image to input into the language model. However, for retrieval tasks, the goal is to distill information from redundant visual tokens into a single [EOS] embedding vector. The computation brought by a large number of tokens grows quadratically, while its marginal contribution to the final semantic quality of the embedding is highly limited. This makes MLLM embedding models difficult to deploy in large-scale, low-latency retrieval systems.

Key Challenge: Prior approaches assume "more visual tokens yield better performance," thus continuously stacking token counts. In reality, retrieval tasks contain significant visual redundancy, creating a severe imbalance between the massive computational cost of long sequences and the limited improvement in embedding quality. Simple token compression damages multimodal understanding, leading to degraded embedding quality.

Key Insight: The key insight of this paper is that the performance loss caused by visual token compression can be compensated for or even reversed through a carefully designed training strategy. Compression is not a "trade-off" but can be leveraged as a "strategic advantage."

Core Idea: The method co-designs a parameter-free visual token compression via bilinear interpolation + pixel unshuffle (reducing tokens to 25%) with a three-stage progressive training pipeline (generative recovery \(\rightarrow\) contrastive learning \(\rightarrow\) MLLM judge fine-tuning). This ensures that the compressed model not only avoids performance drops but also outperforms full-token baselines with much more efficient inference.

Method

The proposed method functions as a comprehensive framework integrating "compression architecture + three-stage training + collaborative reranking," where each component is tightly coupled.

Overall Architecture

The system is divided into two core parts: the visual token compression architecture (InternVL3-VTC) and the three-stage progressive training pipeline, supplemented by a collaboratively trained Reranker.

Visual token compression is parameter-free: bilinear interpolation downsampling followed by a pixel unshuffle operation is sequentially applied to the spatial feature maps output by the ViT, reducing the token count per image tile from 256 to 64 (retaining 25%), which greatly reduces attention computation. However, compression disrupts the expected spatial distribution of visual features in the pre-trained LLM. Therefore, a three-stage training pipeline is required to recover and enhance embedding capabilities:

Stage 1 (Generative Recovery): Autoregressive continuing training is conducted on 32 million multimodal instruction data points to realign the compressed visual features with the LLM semantic space, restoring multimodal understanding and generation capabilities.
Stage 2 (Contrastive Pre-training): Contrastive learning is performed on 16 million multimodal retrieval data points, starting with warm-up on in-batch negatives, and subsequently incorporating Global Hard Negative Mining (sampling 2 hard negatives from candidates ranked 50–100 across the entire dataset).
Stage 3 (Task-Aware Fine-Tuning): Using Qwen3-VL as a judge, a "retrieve-and-judge" process is executed on 1.5 million target training samples. For each query, the Stage 2 model retrieves the top-20 candidates, and the MLLM judge evaluates their relevance to filter high-quality hard negatives (12 per instance) to drive the final contrastive learning.

The Reranker is initialized from the Stage 1 checkpoint and jointly trained on the MLLM judge-tuned data using pointwise (Yes/No binary classification) and listwise (locating the positive sample position among \(M\) candidates) objectives. At inference time, it reranks the top-5 results of the embedder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["ViT Encoding<br/>H×W visual tokens"]
    B --> C["Parameter-free Visual Token Compression<br/>Bilinear Interpolation → Pixel Unshuffle<br/>256 → 64 tokens/tile"]
    C --> D["LLM (InternVL3)<br/>Fuses text + compressed visual tokens<br/>Last token → L2 normalized embedding"]

    D --> E["Three-Stage Progressive Training"]

    subgraph E["Three-Stage Progressive Training"]
        direction TB
        E1["Stage 1: Generative Recovery<br/>32M multimodal instruction data<br/>Autoregressive NTP loss"]
        E1 --> E2["Stage 2: Contrastive Pre-training<br/>16M retrieval data<br/>InfoNCE + Global HNM"]
        E2 --> E3["Stage 3: Task-Aware FB/FT<br/>1.5M MLLM-as-a-Judge fine-tuning data<br/>InfoNCE + 12 HNs/instance"]
    end

    E --> F["Embedding Vectors → Candidate Set"]
    F --> G["Reranker (Same Architecture)<br/>Pointwise + Listwise<br/>Reranks top-5 candidates"]
    G --> H["Final Results"]

Key Designs

1. Parameter-free Visual Token Compression: Avoiding Optimization Difficulties by Substituting Learning with Interpolation

In MLLM embedding models, the number of visual tokens directly dictates the \(O(N^2)\) computational overhead of self-attention. While common learnable compression modules (like BLIP-2's Q-Former, Honeybee) are flexible, introducing extra parameters in retrieval scenarios can lead to training instability or even convergence collapse. This work adopts a purely parameter-free scheme: bilinear interpolation is first applied to the spatial feature maps output by the ViT to downsample \(H \times W\) to \(H' \times W'\), followed by pixel unshuffle (downsampling factor \(\alpha=2\)) to further compress the features into \(H'/\alpha \times W'/\alpha\) tokens (with channel dimensions expanded to \(\alpha^2 C\)). The entire process introduces no learnable parameters and preserves local spatial structures. Ablation studies show that bilinear interpolation outperforms both learnable Conv2d (average 17.9 vs. 61.0) and adaptive average pooling (60.6 vs. 61.0). Conv2d fails because the module and the MLLM struggle to adapt to each other due to cold starts, causing training to collapse; average pooling loses local details, while bilinear interpolation requires no learning and preserves locality, showing a clear advantage in text-to-document retrieval tasks.

For multi-tile scenarios (such as document images), the \(256 \rightarrow 64\) compression is executed independently for each tile. The number of tiles for natural images is limited to 1 (most scenarios) or 4 (documents), resulting in at most 256 visual tokens per image (compared to 1280+ in the baseline), reducing attention computation by 93.8%.

2. Three-Stage Progressive Training: A Coarse-to-Fine Strategy from Generative Recovery to Contrastive Fine-Tuning

Because there is a discrepancy between the compressed visual features and the expected distribution of the pre-trained LLM, performing contrastive learning directly yields poor results (only 1.9 on MMEB). This work designs a three-stage progressive training logic:

Stage 1 (Generative Recovery): Autoregressive next token prediction training is performed on 32 million multimodal instruction data points (including 26.2 million open-source and 5.8 million self-built samples, covering instruction following, captioning, grounding, and classification). The key to this step is to restore the model's ability to interpret and generate from compressed visual features—ablations demonstrate that skipping Stage 1 to directly perform contrastive learning drops MMEB performance from 65.4 to 64.7 and VisDoc from 70.7 to 69.8.

Stage 2 (Contrastive Pre-training with Hard Negative Mining): InfoNCE contrastive learning is performed on 16 million multimodal retrieval data points in two steps—first warming up with simple in-batch negatives, then introducing Global Hard Negative Mining (Global HNM). Specifically, for each query, ranked candidate lists are retrieved from the entire dataset, and after removing the ground-truth positive samples, 2 hard negatives are randomly sampled from positions ranked 50–100. Positions 50–100 are chosen instead of the top-10 to avoid mistakenly treating actual positives as negatives (false negatives).

Stage 3 (MLLM-as-a-Judge Task-Aware Fine-Tuning): Qwen3-VL-8B is used as an external judge to execute a "retrieve-and-judge" process on the target training set. First, the Stage 2 model retrieves the top-20 candidates for each query, and the MLLM judge evaluates the relevance of each pair (by comparing the logits of Yes/No tokens). Positives are added to the augmented positive set (used for reranker training), while top-ranking negatives are used as high-quality hard negatives. Twelve judge-verified hard negatives per instance are incorporated into the InfoNCE calculation. This approach effectively avoids false negatives introduced by rule-based hard negative mining—ablation shows that the MLLM judge scheme consistently outperforms rule-based schemes under the same number of hard negatives (e.g., 68.0 vs 67.0 on MMEB when \(n=12\)).

3. Collaborative Reranker: Jointly Training Pointwise and Listwise Objectives

During inference, a two-stage retrieval strategy consisting of an embedder and a reranker is adopted. The reranker is initialized from the Stage 1 checkpoint and trained on the judge-tuned data of Stage 3. The key innovation is the joint training of two objectives: pointwise (outputting Yes/No for candidate pairs) and listwise (identifying the position \(k\) of the positive sample from \(M \in \{2, 3, 4, 5\}\) candidates). Both loss weights are set to 1. The reranker benefits from the augmented positive and negative sets from the Stage 3 MLLM judge (the augmented positive set contains original ground-truth and new positives identified by the judge), rather than relying solely on the original labels. At inference, pointwise reranking is applied to the top-5 candidates output by the embedder, further improving accuracy.

Loss & Training

• Stage 1: Autoregressive Next Token Prediction (NTP) loss, full-parameter training, \(lr=1\times10^{-5}\), global batch size of 48, 30,000 steps.
• Stage 2 & 3: InfoNCE contrastive loss, temperature \(\tau=0.03\), LoRA rank=16 (all linear layers), \(lr=2\times10^{-4}\).
• Reranker: CE loss for pointwise + listwise, \(lr=4\times10^{-5}\), 2 epochs.
• Hardware: 48 \(\times\) NVIDIA A800 (80GB) GPUs (Stages 2 & 3), 24 \(\times\) A800 (reranker).
• Tile Strategy: Document images MAX_NUM=4, natural images MAX_NUM=1.

Key Experimental Results

Main Results

Table 1: MMEB Benchmark — Universal Multimodal Retrieval (Precision@1)

Model	Backbone	Classification	VQA	Retrieval	Grounding	IND	OOD	Avg
VLM2Vec-V1	Qwen2-VL (2.2B)	59.0	49.4	65.4	73.4	66.0	52.6	59.3
UniME-V2 (E)	LLaVA-OV (8.0B)	65.3	67.6	72.9	90.2	74.8	66.7	71.2
UniME-V2 (E+R)	LLaVA-OV (8.0B)	—	—	—	—	75.2	—	—
Magic-MM-Embedding (E)	InternVL3-VTC (1.9B)	60.9	63.3	72.2	84.6	74.7	59.5	68.0
Magic-MM-Embedding (E+R)	InternVL3-VTC (1.9B)	61.3	67.2	73.5	89.8	75.2	63.9	70.2
Magic-MM-Embedding (E)	InternVL3-VTC (8.1B)	65.0	68.2	74.7	89.6	78.4	63.7	71.9
Magic-MM-Embedding (E+R)	InternVL3-VTC (8.1B)	64.4	71.0	75.7	90.1	78.4	65.9	72.9

At the 2B scale, the Magic-MM-Embedding embedder achieves 74.7 on IND (outperforming the 8B version of UniME-V2 E's 74.8). Its OOD score of 59.5 is highly competitive, despite trailing 8B models. At the 8B scale, the embedder achieves an average score of 71.9, surpassing UniME-V2 (E)'s 71.2, while the E+R version achieves a state-of-the-art score of 72.9. Notably, the 2B embedder (68.0) is close to the UniME-V2 8B E+R (69.0), demonstrating strong efficiency.

Table 2: VisDoc Benchmark — Visual Document Retrieval (NDCG@5)

Model	Backbone	VDRv1	VDRv2	VR	OOD	Overall
GME	Qwen2-VL (2.2B)	86.1	54.0	82.5	43.1	72.7
GME	Qwen2-VL (8.3B)	89.4	55.6	85.0	44.4	75.2
Magic-MM-Embedding (E)	InternVL3-VTC (1.9B)	83.4	53.3	85.6	42.2	72.1
Magic-MM-Embedding (E+R)	InternVL3-VTC (1.9B)	84.4	56.1	87.4	41.8	73.3
Magic-MM-Embedding (E+R)	InternVL3-VTC (8.1B)	86.8	59.6	89.1	42.9	75.5

Even with only 25% of tokens, SOTA results are still achieved on tasks like document retrieval requiring high-resolution precision. The 8B E+R model achieves an overall score of 75.5, surpassing GME 8B's 75.2, despite GME utilizing larger-scale proprietary datasets.

Table 4: Inference Efficiency Comparison (L20 GPU, batch=1, BF16)

Model	Backbone	MMEB Query Latency (ms)	MMEB Candidate Latency (ms)	VisDoc Candidate Latency (ms)
LLaVE	Aquila-VL (2.0B)	162.8	143.0	233.6
UniME-V2	LLaVA-OV (8.0B)	906.9	788.1	1341.1
InternVL3 (Vanilla)	InternVL3 (1.9B)	37.1	29.2	103.6
Magic-MM-Embedding	InternVL3-VTC (1.9B)	29.9	26.1	57.3
Magic-MM-Embedding	InternVL3-VTC (8.1B)	50.9	40.6	94.8

With an average of only 99.6 visual tokens (vs. LLaVE's 3,699 and UniME-V2's 7,371), latency is dramatically reduced. The 2B model's MMEB query latency drops from 162.8ms (LLaVE) to 29.9ms, and the VisDoc candidate latency drops from 233.6ms to 57.3ms.

Ablation Study

Table 5: Ablation on Progressive Training Pipeline & Reranker

Stage 1	Stage 2	Stage 3	Reranker	MMEB	VisDoc
✓	✗	✗	✗	1.9	0.5
✗	✓	✗	✗	64.7	69.8
✓	✓	✗	✗	65.4	70.7
✓	✗	✓	✗	67.1	70.9
✓	✓	✓	✗	68.0	72.1
✓	✓	✓	✓	70.2	73.3

Training only Stage 1 yields poor utility (MMEB 1.9), but skipping Stage 1 to train Stage 2 also degrades performance (\(65.4 \rightarrow 64.7\)), demonstrating that generative recovery is a crucial warm-up step for compressed models. Employing all three stages yields the best performance compared to omitting any. Adding the reranker further boosts performance by 2.2 on MMEB and 1.2 on VisDoc.

Figure 3: Ablation on Visual Token Counts

#Tokens/tile	MMEB	VisDoc	Avg	Attn Computation Reduction
36	62.6	66.8	64.7	98.0%
64	63.2	67.6	65.4	93.8%
100	63.3	68.9	66.1	84.7%
144	63.1	69.1	66.1	68.4%
256 (Vanilla)	63.7	68.5	66.1	0%

The 64-token configuration serves as the sweet spot: it performs just 0.7 percentage points lower on average than using 256 tokens while reducing attention computation by 93.8%. Increasing tokens (100 or 144) offers limited computational savings without showing performance gains, whereas using fewer tokens (36) causes a noticeable performance drop.

Key Findings

• Stage 1 (Generative Recovery) is an indispensable warm-up: Skipping Stage 1 drops performance from 65.4 to 64.7 on MMEB, and from 70.7 to 69.8 on VisDoc. Compression alters the pre-trained visual feature distribution, and generative training successfully realigns these compressed features with the LLM semantic space.
• MLLM judge hard negatives are significantly better than rule-based methods: At the same number of hard negatives (\(n=12\)), the MLLM judge scheme (MMEB 68.0, VisDoc 72.1) consistently outperforms the rule-based approach (MMEB 67.0, VisDoc 70.1). Moreover, the choice of the judge model (Qwen3-VL vs. InternVL3) has little impact on the results (80.4% agreement rate), indicating robustness across judge models.
• Compression is not a trade-off but a strategic advantage: The 64-token model is not only faster in inference, but it also outperforms the full-token UniME-V2 (71.2 / —) at the 8B scale, with 71.9 (embedder) and 72.9 (E+R). This validates that "compression + targeted training" can simultaneously optimize both efficiency and effectiveness.
• 64 tokens represent the optimal balance between efficiency and accuracy: While 36 tokens require fewer calculations, performance drops significantly (\(65.4 \rightarrow 64.7\)); 100/144 tokens perform on par with 256 but offer less pronounced computational savings (\(84.7\% \rightarrow 68.4\%\)).
• Training efficiency is also significantly improved: Under the same contrastive training configuration, the training time for the 2B model drops from 52h 43m to 24h 38m (a 53% reduction) with almost no performance degradation.

Highlights & Insights

• A Paradigm of Parameter-free Compression + Progressive Training Co-design: Most prior studies focus either on token compression (which degrades performance) or on training optimization (while ignoring efficiency). This work demonstrates how they synergize: simple parameter-free compression reduces optimization uncertainty, and targeted training recovers the information loss, achieving a \(1+1>2\) effect.
• "Compressed models require generative warm-up rather than immediate contrastive learning": This is a counter-intuitive but critical finding. It is often assumed that contrastive learning alone can handle post-compression alignment. However, generative recovery in Stage 1 (rather than the retrieval objective) is the crucial first step. Skipping it causes a ~1 point performance drop. This is because compression alters the density and distribution of visual features; hence, generative objectives are better suited for "realigning" visual and language spaces than contrastive objectives.
• An Elegant, Cost-Effective Solution using MLLM-as-a-Judge for Hard Negative Mining: Applying an external MLLM as a judge elegantly solves three primary issues in traditional rule-based hard negative mining: false negatives (highly relevant actual positives being treated as negatives), unpredictable quality of hard negatives, and inconsistent criteria across datasets. Furthermore, the approach is highly robust to the choice of the judge model (80.4% consistency between InternVL3 and Qwen3-VL), lowering deployment barriers.
• The 64-token Sweet Spot Stemming from Retrieval Task Redundancy: The goal of a retrieval task is to compress visual + text information into a single [EOS] embedding vector rather than high-fidelity, pixel-by-pixel reconstruction. Consequently, a massive number of visual tokens yields diminished marginal performance gains. This work systematically pinpoints this sweet spot and provides a reproducible analysis framework for future studies.
• Reranker with Joint Pointwise + Listwise Training: Instead of using traditional pairwise loss alone, this paper jointly trains pointwise (Yes/No predictions) and listwise (locating the positive sample among \(M\) candidates) objectives. These two objectives complement each other and benefit from the judge-augmented positive set in Stage 3, producing better results than training solely with original labels.

Limitations & Future Work

• Performance gap remaining in out-of-domain (OOD) document retrieval: On the VisDoc OOD benchmark, the 8B E+R model reaches 42.9 (lower than GME 8B's 44.4), indicating that compressed models still have room for improvement in cross-domain generalization. Future work could investigate retaining more tokens for document-related tasks or employing hybrid strategies.
• Static compression ratio: Currently, a fixed \(256 \rightarrow 64\) compression ratio (75% reduction) is used, which lacks adaptivity to images of varying complexity. Future work could explore content-based adaptive compression ratios (e.g., 36 tokens for simple scenes and 144 tokens for complex document scans).
• High training budget: Although inference efficiency is significantly improved, the three-stage training pipeline (32M + 16M + 1.5M samples) still requires 48 \(\times\) A800 GPUs, presenting a high barrier to entry for resource-constrained research groups. Full-parameter training in Stage 1 is especially costly; using smaller, high-quality alignment corpora could mitigate this.
• Evaluation limited to the InternVL3 backbone: While InternVL3 is a representative MLLM, the experiments were conducted using only this single backbone. Whether the scheme generalizes to other MLLM architectures like Qwen2.5-VL or LLaVA-OneVision is yet to be validated.
• Additional inference overhead from the MLLM judge stage: The "retrieve-and-judge" process in Stage 3 requires Qwen3-VL-8B to evaluate a massive number of candidate pairs (2.8M pairs). Although this is a one-time offline cost, it might be too expensive for scenarios seeking sample-efficient pipelines. The paper does not thoroughly discuss the quantitative trade-off between judge inference costs and accuracy gains.

Related Work & Insights

• vs. MLLM Embedding Methods (VLM2Vec, UniME, QQMM, LLaVE, etc.): These methods focus on data scale, hard negative strategies, and progressive training to improve MLLM embedding quality, but they all use standard, unmodified MLLM architectures, ignoring the efficiency bottlenecks caused by visual token redundancy. This work is the first to systematically introduce visual token compression to MLLM embeddings, proving that compressed models can outperform full-token baselines while improving efficiency.
• vs. InternVL3 (Vanilla Backbone): The proposed InternVL3-VTC adds only a parameter-free compression module to InternVL3, with the sole difference being the interpolation and pixel unshuffle operations on the ViT feature maps. Comparative experiments show that the compressed model achieves slightly better multimodal understanding performance (86.1 vs. 84.9 on MME), suggesting that compression may act as a regularizer or reduce feature noise.
• vs. General Visual Token Compression Methods (LLaVA-PruMerge, FastV, TokenPacker, etc.): These methods are primarily designed for general image-to-text generation tasks to maintain text generation quality while reducing tokens. This work is the first to systematically investigate compressed tokens on embedding/retrieval tasks, finding that retrieval targets exhibit a higher tolerance for compression because the embedding objective aggregates information into a single [EOS] token rather than reconstructing pixel-level details.
• vs. MLLM-as-a-Judge / Self-Training Methods: Using an external MLLM as a judge to filter hard negatives can be seen as a form of self-distillation and data augmentation. Similar ideas are used in RLAIF-V and self-rewarding frameworks, but this study applies it to hard negative mining for contrastive embedding learning, demonstrating its toughness regardless of the choice of judge model.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Bringing visual token compression into MLLM universal multimodal embeddings and systematically demonstrating that "compression + targeted training" can improve both efficiency and accuracy is a pioneering approach among existing MLLM embedding efforts.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ The experiments cover MMEB (36 subsets), VisDoc (24 subsets), and cross-modal retrieval (5 benchmarks with 3 levels of granularity). It includes comprehensive ablations on token counts, three-stage training, hard negative volumes/types, judge models, LoRA ranks, different compression methods, training efficiency, and vLLM deployment efficiency—virtually exploring all relevant dimensions with rich empirical data.
• Writing Quality: ⭐⭐⭐⭐⭐ Problem definitions are exceptionally clear; the transition from efficiency bottlenecks to the core challenge is highly convincing. The methodology narrative flows smoothly (unfolding through compression \(\rightarrow\) training \(\rightarrow\) reranking), and the ablation designs precisely correspond to the results, analyzed with thorough explanations rather than raw data listings.
• Value: ⭐⭐⭐⭐⭐ The paper directly resolves the practical application barrier of MLLM embedding models: inference efficiency. The results achieve new SOTA across multiple benchmarks, and the efficiency boost (e.g., dropping 2B model latency from 162.8ms to 29.9ms) is highly significant to industries. Its simplicity (parameter-free compression) and reproducible training strategies (using entirely open-source data) make it an excellent baseline for subsequent MLLM embedding research.

Related Papers

• [ICLR 2026] U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning
• [CVPR 2026] MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model
• [ICML 2026] CHARM: 用 Multimodal JEPA + 通道描述做时间序列 foundation embedding
• [CVPR 2026] ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
• [CVPR 2026] Ego: Embedding-Guided Personalization of Vision-Language Models