Skip to content

MR^2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Area: VLM Reasoning
Keywords: Multimodal Retrieval, Reasoning-intensive Benchmark, Multimodal Knowledge Retrieval, Visual Relation Reasoning, Vision-Language Models

TL;DR

MR^2-Bench introduces the first reasoning-intensive multimodal retrieval benchmark spanning 12 diverse sub-tasks across knowledge retrieval, illustration search, and visual relation reasoning, revealing that SOTA embedding models collapse from ~78% on standard benchmarks to under 10% Recall@1, while test-time CoT rewriting and multimodal reasoning rerankers unlock substantial headroom up to +15.22 nDCG@10.

Background & Motivation

Multimodal retrieval has become an indispensable cornerstone of modern artificial intelligence systems, powering retrieval-augmented generation (RAG), visual web search, and autonomous multimodal agentic workflows. Over the past decade, the field has evolved from classic cross-modal alignment (such as text-to-image matching on MS-COCO and Flickr30K) to composed image retrieval and multimodal document retrieval. However, prevailing multimodal benchmarks (such as MMEB, M-BEIR, and ViDoRe) fundamentally remain confined to surface-level semantic correspondence or instance-level matching. Under this conventional paradigm, models succeed largely by spotting superficial lexical overlaps, common object labels, or salient OCR tokens, completely bypassing the need for higher-order reasoning over intricate visual-textual interplay.

This status quo creates a severe mismatch with real-world information demands. Practical scientific inquiry, professional analytical reporting, and autonomous agent tasks constantly require models to interpret complex multi-image document structures, reason through geometric diagrams, and deduce causality that cannot be verified through linguistic tokens alone. Meanwhile, existing multimodal benchmarks almost universally impose an artificial constraint where queries and candidates contain at most a single natural image, ignoring abstract domains like mathematical visual proofs, economic charts, and visual logic puzzles. While text-only retrieval has recently been disrupted by reasoning-intensive benchmarks such as BRIGHT, the multimodal retrieval community has lacked a benchmark where visual signals are truly indispensable and relevance is impossible to resolve without multi-step deduction.

To bridge this fundamental gap, this work shifts the evaluation philosophy from shallow semantic matching to deep, evidence-grounded deduction across heterogeneous modalities. Core idea: introduce MR^2-Bench, the first reasoning-intensive multimodal retrieval benchmark comprising 12 sub-tasks across multimodal knowledge retrieval, visual illustration search, and visual relation reasoning, enforcing strict visual indispensability and adversarial negative mining to evaluate logical, spatial, and causal inference in multimodal systems.

Method

Overall Architecture

MR^2-Bench establishes a comprehensive data curation and evaluation pipeline specifically crafted to eliminate superficial keyword shortcuts and visual surface alignment. Spanning 3 overarching meta-tasks and 12 distinct sub-tasks, the benchmark comprises 1,309 rigorously verified queries and a corpus of 47,742 interleaved text-image paragraphs and high-resolution images.

The overall framework proceeds through multi-source heterogeneous collection, structural taxonomy mapping across knowledge and relation domains, visual indispensability auditing paired with adversarial hard negative synthesis, and free-form multimodal evaluation protocol supporting both single-stage dense retrieval and reasoning-augmented test-time inference. The complete pipeline is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Heterogeneous Multi-Source Data<br/>StackExchange/WorldBank/Proofs/VASR/CSS"] --> B["Three-Tier Reasoning Meta-Task Taxonomy<br/>Knowledge Retrieval/Illustration Search/Relation Reasoning"]
    B --> C["Visual Indispensability Auditing & Adversarial Mining<br/>Image-essential filtering & GPT-5 confusing negatives"]
    C --> D["Free-Form Multimodal Retrieval Benchmark<br/>1,309 Curated Queries / 47,742 Candidate Corpus"]
    D --> E["CoT Rewriting & Reasoning-then-Rank Protocol<br/>Explicit intent decomposition & MLLM scoring"]
    E --> F["Output: Comprehensive Diagnostic Evaluation<br/>nDCG@10 / Recall@K / Fine-grained capability analysis"]

Key Designs

1. Three-Tier Reasoning Meta-Task Taxonomy: Standard single-scenario retrieval fails to probe diverse cognitive depths -> Establish a structured matrix spanning cross-modal deduction, domain illustration discovery, and purely vision-centric puzzles -> Systematically isolate and diagnose model capabilities across orthogonal reasoning dimensions

Conventional multimodal retrieval benchmarks evaluate isolated modalities or uniform cross-modal matching, leaving the specific points of failure obscured. MR^2-Bench structures evaluation across 3 meta-tasks and 12 sub-tasks: 1. Multimodal Knowledge Retrieval: Encompasses 6 scientific and practical disciplinesโ€”Biology, Cooking, Gardening, Physics, Chemistry, and Earth Science. Queries comprise interleaved image-text questions, while the corpus consists of interleaved web paragraphs. Successfully retrieving the ground-truth document requires synthesizing microscopic visual features, chemical configurations, or physical phenomena with underlying theoretical principles. 2. Visual Illustration Search: Encompasses Economics, Mathematics, and Nature. Queries are domain-specific textual hypotheses or formulas, and the candidates are visual artifacts such as statistical charts, geometric proofs without words, or fine-grained anatomical photographs. Models must bridge the abstract symbolic representation and the visual evidentiary diagram. 3. Visual Relation Reasoning: Comprises Spatial geometry, Visual Puzzle (progressive matrices), and Analogy. This meta-task eliminates or sharply reduces linguistic cues, presenting purely visual configurations (such as 8 frames of a 3x3 Raven matrix). Models must deduce implicit transformation laws, attribute permutations, and relational mappings exclusively from vision.

2. Visual Indispensability Auditing & Adversarial Mining: Text shortcuts allow models to bypass vision and trivial negative pools inflate benchmark scores -> Enforce visual-essential filtering and combine human domain validation with model-synthesized hard negatives -> Eliminate lexical shortcuts and push retrievers to their genuine visual-logical limits

To prevent retrievers from exploiting low-level keyword co-occurrence, MR^2-Bench imposes strict quality gates during data collection. For the Stack Exchange knowledge subset, posts are accepted only if the query cannot be understood or answered when the image is removed, if the post gathered at least 5 upvotes, and if the validated answer links to external authoritative literature. Ground-truth documents preserve full multimodal interleaving and undergo expert dual verification to confirm that the text-image passage genuinely supplies the scientific mechanism needed to solve the query.

Furthermore, to defeat superficial semantic clustering, the benchmark constructs two complementary pools of challenging negatives: - Topical Hard Retrieval Negatives: Retrieving documents sharing surface keywords from a massive external corpus, with GPT-5 screening out false negatives. - Model-Synthesized Confusing Negatives: Prompting GPT-5 to generate passages that mimic the topic and vocabulary of the query but contain flawed conclusions or subtle causal inconsistencies. Human annotators subsequently audit and filter all generated candidates, ensuring that only retrievers capable of deep causal discrimination can place true positives at the top ranks.

3. CoT Rewriting & Reasoning-then-Rank Protocol: Single-vector inner products bottleneck complex multi-hop inference -> Formalize test-time chain-of-thought query rewriting and two-stage reason-then-rank MLLM scoring -> Characterize the performance envelope unlocked by test-time reasoning compute in multimodal retrieval

Because dense single-vector embedding models face severe capacity bottlenecks when compressing complex multi-image hypotheses into fixed-dimensional vectors, MR^2-Bench introduces a standardized evaluation framework exploring test-time reasoning compute: - Chain-of-Thought Query Rewriting Protocol: MLLMs (specifically GPT-5) decompose complex visual queries via a structured three-step prompt: identifying fundamental implicit subquestions, determining the requisite physical/theoretical evidence candidates must exhibit, and synthesizing a consolidated search-augmented prompt constrained within 100 words. - Two-Stage Reason-then-Rank Protocol: Candidates retrieved in the top-\(K\) (\(K=20\)) pool are reranked by text rerankers, reasoning-enhanced text models, native multimodal rerankers, and MLLMs prompted to execute explicit step-by-step reasoning before emitting a discrete relevance score \(\in [0, 100]\): $\(\text{DCG}@K = \sum_{i=1}^K \frac{2^{rel_i} - 1}{\log_2(i + 1)}, \quad \text{nDCG}@K = \frac{\text{DCG}@K}{\text{IDCG}@K}\)$ This protocol decouples the baseline representation quality from the test-time reasoning budget, providing a transparent frontier of current capabilities.

A Worked Example

Consider a concrete instance from the Biology sub-task: a user presents an image of two apples fused together into a conjoined double fruit with the question: "How did these apples grow together? I came through this pic while scrolling through facebook... I want a biological answer for this."

The ground-truth document is an academic treatise on floral organogenesis. Notably, the text never mentions the word 'apple' or the phrase 'grow together'. Instead, the document features an anatomical diagram illustrating twin fruit malformation in an entirely different botanical species, accompanied by a detailed explanation of abnormal carpel fusion induced by heat-stressed, defective pollination. - Shallow lexical retrievers: Searching for "apple" or "facebook" completely miss this document, producing an effective zero score. - Surface multimodal retrievers: Although they encode the apple pixels, they retrieve ordinary high-yield orchard pictures due to the lack of developmental biology reasoning. - Reasoning-augmented multimodal systems: Deconstruct the conjoined morphology into "angiosperm carpel fusion defect during fertilization," cross-reference the anatomical diagram across species, and correctly elevate the theoretical explanation to the first rank.

Key Experimental Results

Main Results

MR^2-Bench evaluated 11 representative embedding models across all 12 sub-tasks, encompassing 5 text retrievers (evaluated under both text-only and Qwen2.5-VL-7B detailed caption augmentation) and 6 native multimodal embedding models. The primary metric is \(\text{nDCG}@10\) along with macro-average (Avg.) across all sub-tasks. The raw scores directly quote Table 2 of the paper:

Model Category Method Bio. Cook. Gar. Phy. Chem. Earth. Econ. Math. Nat. Spa. Puzz. Ana. Avg.
Text Embedding BGE-M3 18.79 12.97 12.04 14.52 6.05 16.35 - - - - - - -
(w/ Captions) + Captions 34.19 24.28 17.88 21.24 9.67 25.19 45.46 9.97 23.66 9.48 0.00 3.46 18.71
Text Embedding Qwen3-Emb. 23.77 20.44 12.61 17.13 8.61 19.79 - - - - - - -
(w/ Captions) + Captions 29.97 29.29 18.32 21.46 9.52 23.19 49.44 21.14 26.30 9.11 0.00 4.30 20.17
Reasoning Text Diver-Emb. 27.32 16.94 15.17 18.05 10.06 22.57 - - - - - - -
(w/ Captions) + Captions 38.46 30.87 22.84 23.62 14.46 31.40 54.67 25.91 24.88 8.52 0.00 7.47 23.59
Reasoning Text BGE-Reasoner 29.01 15.37 16.31 21.00 10.62 26.20 - - - - - - -
(w/ Captions) + Captions 42.60 34.40 24.94 25.61 14.31 34.57 54.31 17.16 29.86 5.52 0.00 5.88 25.35
Reasoning Text ReasonIR 29.85 19.72 16.22 21.56 9.83 23.56 - - - - - - -
(w/ Captions) + Captions 44.75 41.91 18.79 27.33 17.45 41.22 64.04 34.49 30.70 11.65 0.00 10.89 25.72
Multimodal Embedding CLIP 32.85 30.57 14.06 14.86 3.50 33.23 12.97 5.64 49.34 20.89 0.19 5.09 18.59
Multimodal Embedding BGE-VL 29.41 18.36 10.50 19.51 7.12 19.73 50.80 14.31 47.97 6.46 0.00 0.75 19.53
Multimodal Embedding GME 34.34 39.50 19.04 19.29 7.73 28.59 36.95 7.19 39.35 15.70 0.22 11.11 21.59
Multimodal Embedding VLM2Vec-V2 39.37 39.38 19.87 20.28 9.03 35.71 51.44 14.16 35.06 13.94 0.62 5.85 23.72
Multimodal Embedding MM-Embed 49.68 52.19 23.67 30.36 17.44 47.51 42.99 21.58 48.41 22.79 0.21 5.93 30.23
Multimodal Embedding Seed1.6-Emb. 40.64 38.12 31.77 27.91 17.80 37.17 56.13 26.10 65.16 17.29 0.93 9.21 30.68

Ablation Study & Reranking Dynamics

To evaluate the impact of test-time reasoning strategies, the table below consolidates GPT-5 Chain-of-Thought Query Rewriting ablation (Table 3) alongside the full suite of reranking models applied over top-\(20\) candidates from the leading Seed-1.6-Embedding retriever (Figure 2 and Table 19):

Category / Model Retriever / Reranker Modality Input Operational Mechanism Average nDCG@10 \(\Delta\) vs. Baseline
Base Retriever Seed-1.6-Embedding Multimodal Dense vector dot-product 30.68 -
Query Rewrite Ablation BGE-M3 (Base) Text + Captions Raw query 18.71 -
Query Rewrite Ablation BGE-M3 + Rewrite Text + Captions GPT-5 CoT rewritten query 23.09 +4.38
Query Rewrite Ablation Seed-1.6 (Base) Native Multimodal Raw query 30.68 -
Query Rewrite Ablation Seed-1.6 + Rewrite Native Multimodal GPT-5 CoT rewritten query 33.87 +3.19
Textual Reranker RankLLaMA-7B Text + Captions Cross-encoder discriminative 27.33 -3.35 (degraded)
Textual Reranker RankLLaMA-14B Text + Captions Cross-encoder discriminative 28.09 -2.59 (degraded)
Reasoning Text Reranker Rank1-7B Text + Captions Test-time reasoning compute 30.21 -0.47
Reasoning Text Reranker RankR1-14B Text + Captions RL-enhanced reasoning 32.28 +1.60
Reasoning Text Reranker ReasonRank-32B Text + Captions RL CoT ranking 32.51 +1.83
Reasoning Text Reranker BGE-Reasoner-32B Text + Captions SFT reasoning reranker 35.08 +4.40
Multimodal Reranker MonoQwen2-VL Native Multimodal Cross-encoder without CoT 30.64 -0.04
Multimodal Reranker Jina-Reranker-m0 Native Multimodal Lightweight 2B dedicated 32.82 +2.14
Multimodal CoT Reranker Gemma3-27B Native Multimodal Prompted Reason-then-Rank 33.51 +2.83
Multimodal CoT Reranker Qwen2.5-VL-72B Native Multimodal Prompted Reason-then-Rank 36.20 +5.52
Multimodal CoT Reranker GLM-4.5V Native Multimodal Prompted Reason-then-Rank 36.37 +5.69
Multimodal CoT Reranker Gemini-2.5-Pro Native Multimodal Long-CoT test-time inference 41.43 +10.75
Multimodal CoT Reranker GPT-5 Native Multimodal Frontier multi-image reasoning 45.90 +15.22

Key Findings

  • Precipitous performance drop for current SOTA models on MR^2-Bench: While Seed1.6-Embedding attains a dominant 77.78% Recall@1 on MMEB, its performance collapses to 9.91% Recall@1 on MR^2-Bench, yielding an average nDCG@10 of only 30.68. Similarly, Diver-Retriever scores 33.90 nDCG@10 on BRIGHT but only achieves 23.59 nDCG@10 on MR^2-Bench with captions.
  • Systemic failure on purely visual relation and puzzle tasks: In the Visual Puzzle (Raven progressive matrices) sub-task, almost all retrievers (including ReasonIR, BGE-VL, and Qwen3) score exactly 0.00% Recall@1, with Seed-1.6 achieving merely 0.93 nDCG@10. In visual analogy and spatial manipulation, models also struggle significantly, showing that current multimodal embedding models rely overwhelmingly on semantic language anchors and lack native perceptual logic.
  • Caption augmentation provides a viable baseline and transfers reasoning power: Augmenting text retrievers with detailed visual captions dramatically boosts their performance across knowledge domains (e.g., ReasonIR reaches 25.72 Avg nDCG@10, outpacing native multimodal VLM2Vec-V2's 23.72). Furthermore, retrievers fine-tuned for reasoning on pure text consistently outperform matching-centric retrievers, showing strong cross-modal generalization of reasoning abilities.
  • Test-time reasoning compute drives dramatic performance leaps: Applying explicit reason-then-rank prompting unlocks massive gains across all baselines. GPT-5 elevates the Seed-1.6 initial retrieval baseline from 30.68 to 45.90 nDCG@10 (an absolute gain of +15.22). Crucially, even lightweight multimodal rerankers like Jina-Reranker-m0 (2B) surpass 32B-scale text-only non-reasoning rerankers, highlighting the indispensable role of joint visual-textual reasoning.

Highlights & Insights

  • Establishment of visual indispensability as a benchmark standard: Unlike prior datasets that indiscriminately append decorative images to text passages, MR^2-Bench strictly verifies that queries become completely insoluble if visual elements are excised.
  • Integration of scientific visual proofs and economic data charts: By curating geometric "Proofs Without Words" and World Bank macroeconomic figures, the benchmark pioneers the formal evaluation of multimodal technical and analytical retrieval.
  • Orthogonal benefits of query rewriting and reason-then-rank pipelines: The empirical results demonstrate that training-free MLLM query rewriting provides an immediate 3โ€“5 point jump across all retrievers, which compounds synergistically when combined with downstream multi-image reasoning rerankers.

Limitations & Future Work

  • Relatively modest query counts in specialized sub-tasks: Curating verified geometric visual proofs and multi-panel World Bank reports with expert double-blind validation is labor-intensive, keeping subsets like Mathematics (86 queries) and Economics (84 queries) modest in scale.
  • Lack of an end-to-end native reasoning retrieval training scheme: The work successfully formulates the benchmark and proves the efficacy of test-time prompting, but how to architect native training objectives and synthesize multi-image CoT supervision for embedding backbones remains an open frontier.
  • vs BRIGHT (Hongjin et al., 2025): BRIGHT established reasoning-intensive text retrieval by harvesting Stack Exchange links; MR^2-Bench extends this paradigm into complex interleaved multimodal documents and purely visual relation tasks, demonstrating that multimodal reasoning induces a much steeper degradation for embedding models.
  • vs MMEB (Jiang et al., 2025) & M-BEIR (Wei et al., 2024): Traditional multimodal retrieval benchmarks are vast but dominated by single-image classification, object detection, or caption matching; MR^2-Bench exposes that models scoring near 80% on these suites collapse to single-digit Recall@1 on multi-image relational reasoning.
  • vs ViDoRe (Faysse et al., 2025): ViDoRe focuses on visual document layout and page OCR parsing; MR^2-Bench specifically tests higher-order scientific causality, geometric transformation, and visual puzzle solving.

Rating

  • Novelty: โญโญโญโญโญ Establishes the first formal, vision-indispensable reasoning benchmark for multimodal retrieval, moving far beyond shallow surface matching.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 11 embedding baselines, conducts caption sensitivity analyses, tests CoT query rewriting, and benchmarks 13 text and multimodal rerankers.
  • Writing Quality: โญโญโญโญโญ Exemplary organization, rigorous task taxonomy, self-contained mathematical definitions, and compelling illustrative case studies.
  • Value: โญโญโญโญโญ Sets a challenging, unsaturated gold standard that will steer future research in multimodal RAG, autonomous vision agents, and System-2 visual reasoning.