MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

Conference: ECCV2026
Paper: ECCV Paper
Project: MultiHaystack
Area: Vision-Language Reasoning
Keywords: multimodal retrieval, unique evidence, retrieval-augmented generation, hard negatives, vision-language reasoning

TL;DR

MultiHaystack separates finding evidence from answering with it using 46,260 heterogeneous candidates and 747 unique-evidence questions, showing that GPT-5 reaches only 51.41% overall answer accuracy with E5-V cross-modality Top-5 retrieval, so reasoning with supplied gold evidence cannot stand in for end-to-end reliability.

Background & Motivation

Vision-language models (VLMs) can perform many fine-grained understanding tasks when an image, video, or document is supplied alongside a question. Real-world knowledge-base question answering must first solve another problem: which file contains the relevant evidence, and what modality is it? A system might understand a manual yet retrieve a visually similar promotional image; fluent downstream reasoning then proceeds without the required evidence. Conventional single-instance visual question answering places the correct context directly in the input, primarily testing understanding rather than finding and then understanding. Retrieval-augmented generation (RAG) requires both stages to succeed, so evaluating only the latter cannot establish system reliability.

Retrieval-oriented benchmarks partially address this gap, but corpus size, modality coverage, and evidence annotation still change the difficulty of the task. Table 1 lists 100–1000 candidates per question for DocHaystack and 10–160 for MMNeedle; small pools can underestimate competition from similar distractors. Knowing in advance that the answer belongs to the video collection removes the need to choose among images, videos, and documents. If several files legitimately support a question, failure to retrieve the annotated file cannot be cleanly equated with failure to retrieve valid evidence. Scale and modality diversity are therefore insufficient by themselves: evidence attribution must be clear enough to locate the origin of an error.

MultiHaystack first constrains each question to a unique supporting item and then places it in a unified pool containing semantically related distractors. It constructs an evaluation using existing multimodal data, model-assisted question generation, and human filtering rather than training a new general reasoning network. Six task types distinguish spatial, temporal, numerical, and factual understanding after retrieval, reducing the risk that aggregate scores hide capability differences. Gold evidence, single-modality retrieval, and cross-modality retrieval provide different conditions for examining the losses introduced by retrieval. Core Idea: combine unique evidence grounding with large-scale heterogeneous distractors, measuring evidence retrieval and evidence-grounded answering separately on the same questions to expose bottlenecks hidden by small, single-modality evaluations.

Method

Overall Architecture

The input is a question and a corpus of images, videos, and documents; the output is an open-ended answer grounded in retrieved evidence. Each question has one uniquely paired correct file, with an additional annotation identifying the relevant page or frame, while retrieval is scored at the file level. Construction collects sources, generates questions, filters ambiguity and questions answerable without evidence, and then adds verified hard negatives. Evaluation ranks files using a vision-language retriever and passes the Top-k files and the question to a multimodal large language model (MLLM). Intermediate artifacts are the uniquely grounded question set, the expanded candidate pool, and a ranked file list for each query. Here, cross-modality primarily means selecting evidence from a mixed candidate pool, not that every question requires jointly using files from multiple modalities. In fact, a single supporting item underpins the benchmark's diagnostic clarity and distinguishes it from multi-document, multi-hop evidence integration. This is a benchmark construction and evaluation paper; the explanation below covers representation, question filtering, distractor enrichment, and evaluation without depicting them as a new model architecture.

Key Designs

1. Heterogeneous sources and unified visual representations: putting different file types into one search space

Image sources include DocHaystack, MMIU, and A-OKVQA; video sources include VideoVista, MMBench-Video, FineVideo, and MVBench; documents come from MINT1T. These sources supply varied visual content and information layouts, yielding 25,652 images, 10,419 videos, and 10,189 documents, or 46,260 items in total. Standalone images are used directly, PDFs are rendered page by page, and videos are uniformly sampled into 8 frames over their full duration. Question generation and retrieval can therefore operate on visual representations across all three file types without requiring a retriever to parse every native format. This common interface has costs: briefly visible events may be missed, and local text or layout details in complex documents may weaken during representation. Consequently, modality differences reflect sampling and information compression as well as high-level semantics.

The paper groups questions by the understanding required by their evidence, rather than only by file type. Visual Parsing and Positioning (VPP, 33 questions) tests spatial relationships; Contextual Understanding (CU, 30) combines visual text or symbols with their surroundings. Video Temporal Reasoning (VTR, 44) concerns motion, order, and state changes; Statistical Reasoning (SR, 321) concerns quantitative relationships in charts and tables. Metadata Identification (MI, 285) locates structured details such as affiliations and timestamps; Factual Knowledge Retrieval (FKR, 34) requires corpus-supported factual answers. These counts sum to 747, but SR and MI dominate, so overall performance must not be read as an equally weighted average of six capabilities. Task-level reporting can reveal whether a retriever with strong aggregate recall remains vulnerable to particular temporal, numerical, or entity constraints.

2. Unique-evidence question filtering: avoiding knowledge shortcuts and ambiguous retrieval labels

GPT-4o generates candidate question-answer pairs from each item's visual representation, averaging approximately 30 questions per item before filtering. The first pass uses GPT-4o and Gemini-2.5-Flash to remove ambiguous questions with multiple valid answers. The second pass uses human review to check for explicit anchors, such as objects, locations, or timestamps, rejecting questions that cannot identify specific evidence. The third pass tests retrieval independence, removing questions answerable without retrieving the supporting item to reduce shortcuts through general knowledge or memorization. The resulting 747 questions remain open-ended yet verifiable, providing traceable grounds for correctness without restricting answers to multiple-choice options. This is a data filtering procedure, not a new training loss for answering models, and it does not guarantee the elimination of all memorization leakage.

Uniqueness means that each question is constrained to one supporting item, not that all questions necessarily originate from distinct files. Page- or frame-level anchors help verify answers, but retrieval counts as successful when the complete file containing the anchor is returned. This distinguishes finding the right file from correctly understanding information inside that file. Page 7 simultaneously reports 433 images, 105 videos, and 209 documents alongside “282 items total,” creating an unresolved counting inconsistency. The first three numbers sum to 747, matching the question count, but the prose alone does not establish whether 282 denotes deduplicated supporting files. This note therefore does not convert that statement into a definite count of distinct evidence items or repeat the accompanying claim of approximately balanced modalities.

3. Hard-negative enrichment: making corpus growth introduce meaningful semantic competition

A question collection containing only supporting items does not adequately simulate locating answers among many similar files. The authors ask GPT-4o to generate query keywords and collect additional candidates through keyword-based web scraping. CLIP similarity and vidore/colqwen2-v0.1 scores then filter candidates for semantic plausibility without redundancy. Human verification checks that these candidates do not contain the correct answer, preventing enrichment from undermining the unique-evidence constraint. Distractors are thus not merely unrelated random noise: they can share an object, event, or visual appearance while lacking a condition essential to the answer. The retriever must preserve constraints such as year, location, or local numerical values instead of satisfying the query through global similarity alone.

The error types in Figure 9 illustrate what this design seeks to expose: a model may favor a salient poster and miss the required video evidence. It may also latch onto the keyword CES while ignoring the specified year, 1974, returning semantically related but temporally incorrect material. To examine scale, the authors retain all positives and add distractors to form 1K, 10K, and full candidate pools. The evidence supporting each question stays fixed while the number of competing candidates changes. Figure 8 reports that E5-V Recall@1 falls from 72.42% at 1K to 40.83% in the full pool, showing why small-pool results cannot be directly extrapolated. Because negatives are selected through keywords and similarity filtering, this experiment measures scaling under that construction process, not a universal scaling law for arbitrary natural corpora.

4. Stage-wise evaluation: separating retrieval failure, understanding failure, and judge error

Retrieval baselines include CLIP, OpenCLIP, Jina-CLIP v1/v2, SigLIP2, Nomic-Embed-Vision, E5-V, and MM-Embed. Single-modality evaluation searches only the corresponding file type, whereas cross-modality evaluation searches the mixed pool. Table 2 reports both settings using the same metrics; Figure 7 additionally controls total pool size for MM-Embed so that modality mixing is not entirely confounded with corpus growth. Recall is averaged over questions: a query succeeds at Recall@k if its uniquely correct file appears among the first k retrieved items. The following is an equivalent notation for that verbal definition, not an additional metric introduced by the paper:

\[ \operatorname{Recall@k}=\frac{100}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathbf{1}\!\left[d^{\star}(q)\in\operatorname{TopK}(q)\right]. \]

Here, \(\mathcal{Q}\) is the evaluation question set and \(d^{\star}(q)\) is its correct file; results are expressed as percentages. Because scoring is file-level rather than page- or frame-level, a hit does not establish precise localization of the answer inside the file. The main answering experiment fixes E5-V Top-5 retrieval so that different MLLMs use the same retrieval frontend. Models include Ola-7B, InternVL-3-8B, Qwen2-VL-7B, Gemini-2.5-Flash, and GPT-5, with no new training objective introduced on this benchmark. Complete retrieved files are provided in principle; context limits or video restrictions trigger a unified preprocessing policy, such as prioritizing content around a matched page or frame. Thus, using a complete file as evidence does not mean every model receives every original file in full without information loss.

Gold Top-1/5 directly supplies an answer-containing file to examine performance with correct evidence and must not be confused with actual retrieval Top-1/5. GPT-4o-mini judges answers under a fixed rubric; answer accuracy is the fraction judged correct, not evidence recall. For each model, the authors sample 30 non-overlapping question-answer pairs for MTurk human annotation and compare these labels with the automated judge. The lowest Cohen's \(\kappa\) in Table 6 is 0.865, supporting high agreement on a small sample, not establishing that all 747 questions were human-audited. This protocol compares stages, but actual answer accuracy cannot mechanically be written as retrieval recall multiplied by gold-evidence accuracy. The difficulty of retrieval misses, additional distracting evidence, and answering without evidence can all invalidate that multiplication.

Key Experimental Results

Main Results

The following excerpt from Table 2 (page 8) reports overall file-level recall in %; cross-modality uses the full mixed pool, whereas single-modality searches only the corresponding file type.

Retriever	Cross-modality R@1	Cross-modality R@3	Cross-modality R@5	Single-modality R@1	Single-modality R@5
CLIP	26.10	37.08	41.10	36.28	51.94
SigLIP2	40.96	51.27	57.03	51.81	67.87
E5-V	40.83	58.90	66.00	50.87	75.64
MM-Embed	38.42	49.53	57.30	51.41	72.42

E5-V has the strongest cross-modality R@5, but the highest cross-modality R@1 in Table 2 is SigLIP2's 40.96%, not E5-V's 40.83%. The single- to cross-modality R@1 gap is 10.04 percentage points for E5-V and 10.85 for SigLIP2; the prose claim of a drop exceeding 40 points conflicts with the table.

The following excerpt from Table 4 (page 10) fixes E5-V Top-5 retrieval and reports answer accuracy (%), not retrieval Recall@5.

Answering model	Cross-modality video	Cross-modality image	Cross-modality document	Cross-modality overall	Single-modality overall
Ola	14.29	20.09	36.36	23.83	34.00
InternVL-3	17.14	29.33	49.28	33.29	39.89
Qwen2-VL	16.19	16.86	19.62	17.54	23.29
Gemini-2.5-Flash	52.38	35.10	56.94	43.64	50.87
GPT-5	60.00	43.19	64.11	51.41	59.84

GPT-5 overall accuracy falls from 59.84% with single-modality retrieval to 51.41% with cross-modality retrieval, a gap of 8.43 percentage points. Page 3 and Figure 2 report 80.86% for Gold Top-1; its gap to cross-modality Top-5 is 29.45 percentage points, although the evidence composition differs between these conditions.

Ablation Study

The following excerpt from Table 7 (page 15) analyzes advanced retrieval pipelines rather than ablating modules of a newly proposed model; all metrics are overall recall (%).

Config	Retrieval condition	R@1	R@3	R@5
E5-V	Single-modality reference	50.87	71.08	75.64
E5-V	Cross-modality	40.83	58.90	66.00
E5-V + Refined Query	Cross-modality	43.78	60.91	68.81
E5-V + MMSearch	Cross-modality	44.58	62.78	70.68
VisRAG	Cross-modality	45.38	54.62	58.37

Adding MMSearch raises E5-V cross-modality R@5 by 4.68 percentage points, but it remains 4.96 points below single-modality E5-V. VisRAG exceeds base E5-V at R@1 but trails it at R@5, showing why performance at different retrieval depths cannot be summarized as simply stronger or weaker.

Key Findings

• Retrieval depth and answering budget must be considered together: additional candidates can cover the correct evidence while exposing the answering model to more distractors.
• The matched-size comparison in Figure 7 supports an independent effect of modality heterogeneity, but does not causally attribute every loss to embedding misalignment.
• Table 5 reports GPT-5 at 43.61% on SR and 58.95% on MI; these are end-to-end task scores, not pure reasoning accuracies conditioned on successful retrieval.

Highlights & Insights

• Clear diagnostic units: page/frame annotation and file-level retrieval scoring serve different purposes. This makes finding the correct file but answering incorrectly an analyzable event rather than leaving only one aggregate score.
• Negatives need not be unrelated: candidates close to a query but lacking its answer expose omitted constraints. This is more diagnostic than enlarging a corpus mostly with irrelevant examples.
• Engineering implication: retrieval hits, internal evidence localization, and final answers should all be examined. Replacing the answering model alone cannot guarantee fixes for upstream modality bias or lost temporal constraints.

Limitations & Future Work

• Coverage boundary: the 747 questions emphasize SR and MI, and each has one supporting item; the benchmark does not directly measure multi-hop RAG that requires combining several files.
• Construction and representation bias: model-assisted generation, specific similarity filters, and 8-frame video representations may shape question and failure distributions. This is a reader assessment of methodological boundaries, not a bias quantified by the paper.
• Limited judge validation: human checks on 30 questions per model do not guarantee equivalent agreement at larger scale or across answer styles, and cannot exclude systematic automated-judge bias.
• Source and extraction uncertainty: the page 7 evidence counts remain unresolved; Table 4 calls its gray references Recall@5 although the text describes answering performance; the text near Figure 5 calls 64.11% overall accuracy, while Table 4 places it in the document column. This note follows the table interpretation.
• Research directions: the authors suggest modality-aware representations preserving layout and time, plus reasoning-state-guided query rewriting, re-ranking, and evidence expansion. The reader additionally recommends reporting cost and answer performance split by retrieval success for such methods.

Related Work & Insights

• vs DocHaystack: the earlier benchmark focuses on vision-language reasoning over document collections, whereas MultiHaystack mixes images, videos, and documents; the added challenge is evidence-type selection and scale, not just longer inputs.
• vs MMNeedle / MM-NIAH: this benchmark emphasizes a unique file-level support item and a larger heterogeneous pool. That control improves diagnostic clarity but does not cover every open-world case with several valid evidence sources.
• vs E5-V / MMSearch / VisRAG: these are evaluated representations or retrieval pipelines, whereas MultiHaystack is the testbed. Modest improvements in Table 7 motivate preserving fine-grained constraints, not a universal conclusion that agentic retrieval is ineffective.

Rating

• Novelty: 4/5, combining a large heterogeneous pool, unique evidence, and stage-wise diagnostics rather than introducing a training algorithm.
• Experimental Thoroughness: 4/5, covering retrievers, answering models, tasks, corpus sizes, and advanced pipelines, with limited question balance and human verification scale.
• Writing Quality: 3/5, a clear central question, but inconsistent counts, metric captions, and result descriptions hinder verification.
• Value: 4/5, useful for diagnosing upstream multimodal RAG bottlenecks; these are reader assessments, not experimental metrics reported by the paper.