Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings¶

Conference: ECCV2026
Authors: Haonan Jiang, Yuji Wang, Yongjie Zhu, Xin Lu, Wenyu Qin, Meng Wang, Pengfei Wan, Yansong Tang
Official Paper: 3466
PDF: Full paper
Area: Multimodal Reasoning
Keywords: multimodal embeddings, traceable evidence, retrieval alignment, reinforcement learning, chain-of-thought

TL;DR¶

Embed-RL uses retrieval feedback from a frozen embedder to train a separate reasoner and supplies the embedder with image boxes, video keyframes, and text keywords from its chain-of-thought, allowing Embed-RL-4B to reach 68.1 overall on MMEB-V2, 3.6 points above UME-R1-7B.

Background & Motivation¶

Universal multimodal embeddings aim to represent text, images, videos, and visual documents as directly comparable vectors, enabling one model to support image-text retrieval, answer retrieval, and content localization. CLIP-style dual encoders handle relatively direct semantic matching well, but can miss what an instruction actually asks to match when inputs interleave images and text. Methods such as VLM2Vec therefore exploit the understanding capabilities of multimodal large language models and shape their vector spaces through contrastive learning. However, extracting the hidden state at the end of an input does not fully exploit the model's ability to analyze a task step by step.

Adding chain-of-thought does not automatically solve this problem. UME-R1 combines generation and embedding, but this paper argues that jointly optimizing generative and contrastive objectives can produce conflicting gradients. TTE separates the reasoner from the embedder and uses generated context to improve retrieval, yet does not directly train the reasoner with embedding-quality feedback. A correct, detailed explanation may consequently fail to express the modification or correspondence that retrieval requires. Moreover, compressing visual evidence entirely into text loses concrete locations and critical moments, leaving the embedder to rediscover evidence within redundant inputs.

Embed-RL retains the decoupled architecture but moves the reasoner beyond generic description. It generates structured trajectories that support evidence extraction from the original input, while the embedder and an independent discriminator evaluate retrieval outcomes and correspondence between trajectories, respectively. Core idea: make reasoning provide multimodal evidence that can be read again, and use retrieval rewards to learn which evidence and explanations help distinguish positives from similar negatives.

Method¶

Overall Architecture¶

The system contains a Reasoner and an Embedder. The former generates evidential Traceability CoT, or T-CoT; the latter jointly encodes the original input, reasoning, and extracted visual evidence into vectors for cosine-similarity retrieval. Training proceeds through “Evidence-Based T-CoT” and “Embedder Filtering and Freezing,” followed by “Dual-Guidance EG-RL” to update the reasoner. This order ensures that retrieval rewards come from an embedder that already has discriminative capabilities and remains fixed during reinforcement learning.

The diagram summarizes the path from training to use. The independent discriminator belongs to reward computation within Dual-Guidance EG-RL; it does not replace the embedder as a ranking module during final retrieval.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal queries and positives"] --> B["Evidence-Based T-CoT"]
    B --> C["Embedder Filtering and Freezing"]
    C --> D["Dual-Guidance EG-RL"]
    D --> E["Optimized reasoner<br/>and frozen embedder"]
    E --> F["Vector-similarity retrieval"]

Key Designs¶

1. Evidence-Based T-CoT: point reasoning to cues that can be read again in the original input

T-CoT is not an unrestricted explanation. It is a structured output containing <thinking>, <rethink>, and <answer> in that order. The first part analyzes the task and extracts modality-specific cues: text_keywords records text keywords, bbox_2d specifies two-dimensional image boxes, and key_frames identifies important video frames. The second part revisits how these cues relate to the retrieval request, and the final part summarizes the information to match. The important change is not the tags themselves, but the shift from describing what is visible to explaining what matters for matching the target.

The coordinates and frame references also drive actual image-region cropping and keyframe extraction, returning visual evidence to the embedder rather than merely appending coordinates as text. The original input is retained: the system combines it with structured reasoning and extracts the final-layer hidden state of the trailing special token <emb> as the representation. This preserves global context while emphasizing local objects, spatial relationships, or the moment of a critical action; text keywords highlight retrieval-relevant content in long descriptions. The mechanism can reduce the influence of irrelevant background, but the correctness of predicted boxes and frames still depends on the reasoner. It provides no formal guarantee that the evidence is valid.

2. Embedder Filtering and Freezing: first establish a retrieval space that can evaluate reasoning

An unreliable embedder cannot tell the reasoner which explanation is better for retrieval. The authors first construct query-positive pairs from image tasks in MMEB-train, video-language data in LLaVA-Hound, and visual-document data in ViDoRe and VisRAG, then generate T-CoT for both sides. A judgment prompt checks whether the query and positive trajectories are clearly irrelevant to the task or contradictory. Only pairs that pass this check enter contrastive learning. Here, retained samples receive No to the question of irrelevance or contradiction; No must not be mistaken for a negative-pair label.

The initial pool contains 2.22 million samples, of which 1.83 million remain after filtering. InfoNCE trains the embedder to bring queries and positives together and separate in-batch negatives while reading the original content and T-CoT; the embedder is then frozen. Difficult filtered-out examples are used for reinforcement-learning exploration. The text describes this subset as approximately 20% of “filtered-out samples,” whereas the diagram shows an 80% contrastive-learning and 20% reinforcement-learning split. The denominator is ambiguous, so an exact reinforcement-learning sample count cannot be inferred.

Freezing does not imply that further embedder training could never help. Its role here is to hold the evaluation standard fixed: reinforcement learning changes only the reasoner's trajectories, avoiding reward drift from simultaneous updates to both networks and keeping generative optimization away from the trained embedder parameters. The reasoner can thus learn to change its focus regions, extracted cues, or expression of retrieval intent without simultaneously reshaping the entire vector space.

3. Dual-Guidance EG-RL: check both retrieval outcomes and query-target trajectory correspondence

The format reward checks the three-part structure and required modality cues, assigning 1 for compliance and 0 otherwise. The outcome reward uses the frozen embedder to compare T-CoT-conditioned query and target representations. Only when the positive ranks among the top \(k\) candidates does its similarity advantage over in-batch negatives contribute a reward. Negative similarities receive temperature-controlled softmax weights rather than equal weights, emphasizing confusing negatives. The mechanism can be expressed as:

\[ R_{\mathrm{outcome}}=\mathbf{1}[\operatorname{rank}(t^+)\leq k]\left(s(q,t^+)-\mathbb{E}_{\tau}[s(q,t^-)]\right). \]

Here, \(s\) is cosine similarity between normalized embeddings, and \(\mathbb{E}_{\tau}\) is the temperature-weighted average over in-batch negative similarities. This expression is reconstructed from the definitions accompanying the original equation. Some operators are missing from the local text extraction, so the PDF equation remains authoritative; the cache does not establish the numerical values of \(k\) or the temperature. The reward is computed in both query-to-target and target-to-query directions rather than optimizing only one side of retrieval.

The process reward introduces an independent pretrained vision-language discriminator. Given a query T-CoT, it selects the best match from shuffled candidate target T-CoTs, including multiple rollouts of the same positive pair and negative trajectories from other samples. Selecting any member of the ground-truth positive set earns 1; otherwise, the reward is 0. The reverse direction is evaluated as well. Shuffling mitigates position bias. “Process” here means matching the completed intermediate reasoning trajectories, not assigning correctness labels to every token or reasoning step.

The three rewards are added with nonnegative weights, and GRPO updates the reasoner. Each group contains 8 sampled trajectories; reward normalization using the group mean and standard deviation produces advantages, while clipped importance ratios and a KL constraint relative to a reference policy limit updates. The outcome reward requires representations that retrieve positives, the process reward requires corresponding query and target explanations, and the format reward ensures evidence is readable by downstream processing. These address distinct failure modes: correct formatting or discriminator selection is not equivalent to retrieval success.

A Worked Example¶

Figure 1 considers a request to replace the drink in a cola advertisement with a real orange soft-drink bottle and retrieve an image satisfying that change. Before reinforcement learning, the reasoner may simply check whether the source contains an orange bottle and return an empty box when none is found. That response may be correct about the source image, yet it misses that modification-based retrieval seeks a changed target.

The optimized trajectory first identifies the source cola-glass region to replace, then states that the target should contain an orange soft-drink bottle in a similar advertising or product-display context. The crop exposes the subject of the modification, the original image preserves the scene, and the final explanation specifies the direction of change. Together, these form the query representation. The improvement is coordination between evidence and retrieval intent, not treating an absent target object as an observed fact.

Loss & Training¶

The embedder uses Qwen3-VL-2B or Qwen3-VL-4B with LoRA, DeepSpeed ZeRO-2, and a sub-batch strategy for 2 epochs of contrastive learning. The learning rate is \(10^{-4}\), weight decay is 0.01, and batch sizes are 512 and 256 for the 2B and 4B models, respectively. The separate Qwen3-VL-8B reasoner is trained with GRPO for 1 epoch, using batch size 256 and learning rate \(3\times10^{-6}\).

Consequently, the 4B in Embed-RL-4B describes the embedder, not the parameter count of the entire system. The main paper does not provide all coefficients needed to reproduce the reward settings or fully report end-to-end latency. Claims about limited compute should be interpreted within these boundaries.

Key Experimental Results¶

Main Results¶

MMEB-V2 contains 78 datasets: 36 image, 18 video, and 24 visual-document datasets. Image and video tasks use Hit@1, which tests whether the first retrieved result is a hit; visual-document tasks use NDCG@5 to measure ranking relevance among the first 5 results. Higher scores are better throughout the table. The overall score is the paper's cross-task aggregate, not a uniform accuracy metric. Values come from Table 2.

Model	Image overall ↑	Video overall ↑	Visual-document overall ↑	All tasks ↑
GME-7B	56.0	38.6	75.2	57.8
VLM2Vec-V2-7B	68.1	36.4	69.3	61.2
UME-R1-7B	71.3	47.5	67.1	64.5
Embed-RL-2B	69.2	52.1	74.1	66.8
Embed-RL-4B	70.1	53.0	74.7	68.1

Relative to UME-R1-7B, Embed-RL-4B gains 3.6 points overall and 5.5 points on video, but scores 1.2 points lower on images. Its visual-document aggregate also falls below GME-7B's 75.2. Thus, the best overall result does not mean the best result on every modality or task. More specific strengths include 91.4 on image grounding and 67.1 on visual-document OOD, the latter exceeding GME-7B's 44.4 in Table 2.

UVRB covers 16 datasets. The following mAP scores come from Table 1, with higher being better; CG, FG, and LC denote coarse-grained, fine-grained, and long-context retrieval. Its metrics and task composition differ from MMEB-V2, so scores should not be compared directly across benchmarks.

Model	CG mAP ↑	FG mAP ↑	LC mAP ↑
GME-7B	51.8	50.7	78.8
GVE-3B	55.2	54.1	76.4
Embed-RL-2B	59.1	54.6	86.9
Embed-RL-4B	60.7	55.6	86.1

Ablation Study¶

The table combines Tables 3 and 4, both using Embed-RL-2B on MMEB-V2. Overall changes are direct score differences from the full model, not relative percentages. The effects of separate removal experiments should not be added to explain the total gain.

Config	Image overall ↑	Video overall ↑	Visual-document overall ↑	All tasks ↑	Overall change
Full model	69.2	52.1	74.1	66.8	0.0
Without EG-RL	68.0	50.1	72.7	65.3	-1.5
Without negative weighting	68.9	51.7	73.9	66.5	-0.3
Without process reward	68.3	51.3	73.5	66.0	-0.8
Without outcome reward	68.1	51.2	73.1	65.8	-1.0
Without reasoning, answer retained	67.9	50.5	73.1	65.5	-1.3
Without multimodal cues	68.1	51.4	73.3	65.8	-1.0
Raw input only, no T-CoT	60.4	43.7	72.4	60.2	-6.6

The ablation prose calls 69.2→67.9 and 52.1→50.5 changes in image grounding and video moment retrieval, respectively, but Table 4 labels these columns as image and video aggregates. This note follows the table headers rather than attributing those numbers to the specific subtasks.

Key Findings¶

T-CoT as a whole matters more than any single reward component: raw input alone loses 6.6 points overall, with video falling from 52.1 to 43.7, whereas removing reinforcement learning loses 1.5 points. This supports first supplying useful context and then aligning it to retrieval, rather than attributing every improvement to reinforcement learning.
Both process and outcome rewards help, but removing the outcome reward causes a larger overall loss. These ablations do not demonstrate that the discriminator judges every reasoning step correctly, nor do they establish independently additive contributions.
Figure 5 qualitatively shows more boxes for images and documents but fewer video keyframes after reinforcement learning, alongside better retrieval. Evidence selection may therefore adapt by modality rather than obeying a rule that more evidence is always better. The extracted figure text is unsuitable for reading precise counts.

Highlights & Insights¶

Traceable evidence is not merely an explanation for readers: it determines which regions and frames the embedder reads. Reasoning changes the organization of visual input rather than only adding text.
The frozen embedder serves as an evaluator closely related to the deployment objective. Reasoning is assessed by whether it makes positives more distinguishable in vector space, not merely whether its explanation resembles a correct answer.
Bidirectional rewards constrain trajectories on both the query and target sides. This avoids optimizing query explanations while leaving candidate representations disconnected from them.

Limitations & Future Work¶

System cost cannot be inferred from the 2B or 4B embedder alone: it also includes an 8B reasoner, evidence extraction, and an independent discriminator during reinforcement learning. The main paper lacks complete measurements of throughput, index-construction cost, and query latency, which are needed to substantiate efficiency advantages.
Boxes, keyframes, and trajectory matching can all be wrong, and the independent discriminator is not human ground truth. The paper does not systematically quantify how incorrect evidence propagates into embeddings or provide a sufficiently detailed distribution of failure cases.
The local cache contains the complete main paper and references, but not the cited supplementary material. Ambiguities in the reinforcement-learning data split, missing reward weights, and damaged equation extraction leave reproducibility gaps. Verification should clarify these details rather than fill them in from diagrams or assumed defaults.

Compared with VLM2Vec / VLM2Vec-V2: these primarily use contrastive training to turn multimodal large language models into embedders. Embed-RL adds generated evidence context that can be optimized through feedback. The representation gains also introduce an additional reasoning path.
Compared with UME-R1: both investigate generative reasoning for embeddings. Embed-RL separates generative-policy updates from embedder-parameter updates and explicitly incorporates visual evidence. Its experiments support higher aggregate performance, not superiority on every image task.
Compared with TTE: both decouple the reasoner and embedder. Embed-RL additionally optimizes the reasoner with retrieval feedback rather than only consuming pregenerated context. The key distinction is whether context is optimized for the retrieval objective, not whether chain-of-thought exists.

Rating¶

Novelty: 4/5 — Combines evidence-based reasoning, frozen-embedder rewards, and trajectory matching into a retrieval-focused training framework.
Experimental Thoroughness: 4/5 — Two benchmarks and component ablations provide broad coverage, but full-system cost and error-propagation analyses are missing.
Writing Quality: 3/5 — The main argument is clear, although the data split and ablation prose contain ambiguities requiring verification.
Value: 4/5 — Useful for universal embeddings that require fine-grained multimodal evidence; deployment decisions still need stronger efficiency measurements.