MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: multimodal document retrieval, visually rich documents, enrichment-augmented indexing, BM25F, visual late interaction
TL;DR¶
Addressing the excessive serving-time storage and computational overhead of visual multi-vector retrievers like ColPali and ColQwen2.5, MIDR introduces a training-free enrichment-augmented indexing framework that moves multimodal reasoning to offline ingestion via an extract–verify–refine loop, materializing visual layout evidence into structured text fields to enable fast, lightweight BM25F and dense text-centric serving.
Background & Motivation¶
In enterprise reports, regulatory filings, financial statements, slide decks, and scientific articles, crucial factual evidence often resides outside ordinary running text. High-value data is routinely encoded in multi-column tables, trend charts, infographics, captions, and visual groupings: a financial metric depends on row and column headers, a scientific trend hinges on axes and legends, and slide takeaways rely on spatial layout. Plain optical character recognition (OCR) fundamentally fails on these visually rich documents because it forces two-dimensional spatial structures into flattened, linear character streams, corrupting cell associations, stripping geometric groupings, and completely discarding non-textual graphical signals. When downstream retrieval-augmented generation (RAG) pipelines ingest these flattened representations, the retriever fails to surface the relevant pages, causing cascading hallucination and answering failures.
To address this representation gap, the prevailing state-of-the-art paradigm—epitomized by ColPali and ColQwen2.5—abandons text-only representations and retrieves directly over rendered page images. By encoding each page into hundreds of patch-level multi-vector representations and computing MaxSim late-interaction scores at query time, these visual retrievers successfully preserve visual and layout cues. However, this design choice keeps heavy multimodal computation directly on the query-time serving path. Storing multi-vector representations inflates index memory by tens of times compared to lexical indexes, and executing late-interaction scoring across large candidate pools introduces substantial query latency. In real-world enterprise deployments where indexes are queried millions of times and agentic workflows issue multiple retrieval calls per user request, sustaining large visual multi-vector indexes on the serving path incurs prohibitive GPU hardware costs.
The central insight of this paper is that visually rich document retrieval undoubtedly requires multimodal understanding, but that understanding does not have to happen during query execution. Properties such as entities, claims, quantities, table structures, and chart trends are query-independent document characteristics that can be inferred once during offline ingestion. Multimodal Large Language Models (MLLMs) now possess the visual reasoning capabilities needed to interpret complex page images and convert layout-dependent evidence into high-fidelity structured text. Core idea: shift multimodal reasoning entirely from the recurring query-time serving path to the amortized offline ingestion phase, using an extract–verify–refine MLLM loop to produce grounded textual enrichment fields that can be served via standard BM25F and dense text indexes at roughly 9× smaller memory and 2× lower latency while matching visual late-interaction accuracy.
Method¶
Overall Architecture¶
MIDR (Multimodal Indexing for Document Retrieval) decouples multimodal document retrieval into an offline enrichment and indexing stage and an online text-centric retrieval stage. During offline ingestion, documents are split into rendered page images and raw extracted page text. A document-level MLLM pass first analyzes the first five pages to extract global context, including document type, overarching focus, and salient entities. These global signals condition the page-level enrichment process, helping resolve repeated acronyms and domain-specific references. Each page then undergoes an extract–verify–refine loop: an extractor generates a structured multi-field draft from the image and text; a deterministic postprocessor normalizes tags and removes duplicates; a verifier audits the draft against the original page image and extracted text across five grounding dimensions; and a refiner surgically edits only the flagged fields if inconsistencies are detected. Finally, the verified textual fields are indexed into a BM25F inverted index and embedded via a lightweight dense encoder (EmbeddingGemma) with mean pooling. At query time, incoming text queries are scored purely against lexical and dense textual indexes and fused via Reciprocal Rank Fusion (RRF), completely eliminating the need for page-image processing, visual multi-vector indexes, or GPU late-interaction scoring.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
A["Visually Rich Documents<br/>Rendered Page Images + Extracted Text"] --> B["Document-Level Context Extraction<br/>Summarize Global Topic & Salient Entities"]
B --> C["Multimodal Page-Level Extraction<br/>Conditioned Multi-Field Structured Draft"]
C --> D{"Five-Point Fact Verification<br/>Audit Against Image & Text for Grounding"}
D -->|Issues Flagged| E["Surgical Field Refinement<br/>Selectively Patch Unsupported Fields"]
D -->|Zero Issues| F["Normalized Structured Record<br/>Deduplicate QA & Align Layout Flags"]
E --> F
F --> G["Text-Centric Index Construction<br/>BM25F Sparse Index + Dense Mean Pooling"]
G --> H["Online Text-Only Serving<br/>Reciprocal Rank Fusion (RRF) for Top-K"]
Key Designs¶
1. Document-Level Context Extraction: Resolving Local Page Ambiguity with Global Priors
Single pages in complex technical reports or financial filings rarely exist in isolation; they frequently feature ambiguous abbreviations, standalone tables, or ungrounded pronouns that confuse single-page vision-language models. To supply essential global context without ingesting an entire lengthy document at once, MIDR runs a lightweight document-level pass over the first five pages of extracted text. This pass extracts three query-independent fields: the document genre (document_type), the primary purpose or topic (document_focus), and a set of salient entities (main_entities, including organizations, products, regulations, or systems). These fields are prepended to the prompt for all subsequent page-level enrichment calls, providing a stable interpretive frame. Furthermore, these document-level fields are replicated into the search index for every page in the document, ensuring that localized pages can still match queries targeting global document identity and broader topical scope.
2. Hierarchical Multi-Field Enrichment Schema: Functional Specialization Across Retrieval Modalities
Rather than asking an MLLM to generate a single unstructured page description that blurs distinct search signals, MIDR structures page knowledge into a typed multi-field schema with precise retrieval roles. The schema cleanly separates indexed retrieval fields from routing and control fields:
- Synthetic Question-Answer Pairs (Coarse QA & Fine QA): Serve as the primary lexical bridge between user queries and document content. Coarse QA (3–5 pairs) captures high-level user intents and summaries, while Fine QA (5–15 pairs) generates specific, cell-level questions. Crucially, prompt guidelines mandate that answers must include both the value and its full semantic context (e.g., rejecting bare numbers like "42%" in favor of "Net revenue grew 42% year-over-year to $12.4B in Q4 2024").
- Table and Chart Summaries (Table & Chart Summaries): Expose structural relationships that OCR flattens. Table summaries explicitly textualize row and column headers, units, and comparative cell relations; chart summaries convert visual axes, legends, trends, and plotted quantities into queryable prose.
- Keyphrases and Topic Tags: Extract compact domain-specific descriptors and entity–metric triples tailored for dense semantic embedding alignment.
- Routing and Quality Controls: Control fields such as layout (identifying genuine tables/charts rather than plain bulleted lists) and signal_quality (marking cover pages, blank pages, and tables of contents) prevent decorative pages from generating noisy index content.
3. Extract–Verify–Refine Loop: Preventing Offline Hallucination Cascades in Long-Lived Indexes
Because index-time errors persist permanently and degrade retrieval for thousands of future queries, uncontrolled generative document expansion poses severe reliability risks. MIDR mitigates this with a conservative three-stage loop. After initial multi-field extraction, a deterministic postprocessor validates formatting and strips duplicate questions. Then, a dedicated verifier call audits the draft against the original page image and extracted text across a five-point checklist: layout consistency, factual grounding (verifying numbers, dates, and units), internal consistency (preventing contradictory answers), answer quality (flagging uncontextualized bare values), and completeness (checking for omitted table cells). Only when zero issues are identified (is_consistent=true) is the draft accepted. If discrepancies are found, a refiner is invoked to patch strictly the flagged fields while preserving the verified remainder. Across the benchmark, this verification gate triggered refinement on 9.6% of pages (peaking at 52.3% on the complex HR domain), enforcing high factual precision at minimal extra compute.
4. Hybrid Text-Centric Serving: Eliminating Serving-Time Visual Multi-Vector Overhead
Once verified, the enrichment fields and original page text are indexed exclusively into standard text retrieval infrastructure, bypassing image processing at query time entirely. For sparse retrieval, MIDR indexes each field separately in a BM25F engine, allowing uniform or role-based weighting without stemming on enrichment fields to preserve named entities. For dense retrieval, MIDR embeds the original text and each enrichment field separately using EmbeddingGemma (a compact 308M-parameter multilingual bi-encoder) and combines them into a single L2-normalized vector via mean pooling:
The resulting vectors are indexed in a standard FAISS IndexFlatIP. At query time, lexical and dense candidate rankings are fused using Reciprocal Rank Fusion (RRF):
with the standard constant \(k=60\). This architecture delivers sub-15ms response times on commodity CPU/GPU hardware without allocating memory for visual patch tokens.
Benchmark & Evaluation Protocol¶
The framework is rigorously evaluated on ViDoRe V3, the standard multimodal document retrieval benchmark spanning 2,099 queries over 16,867 pages from 184 documents across 7 specialized domains: 1. Five English-Document Domains: Computer Science (CS), Finance, Human Resources (HR), Industrial, and Pharmaceuticals (1,489 queries, 12,968 pages, 101 documents). 2. Two French-Document Domains: Energy and Physics (610 English queries, 3,899 French pages, 83 documents), serving as a cross-lingual stress test where English enrichments must bridge language mismatch. The primary evaluation metric is nDCG@10 computed via official qrels and ir-measures. Systems compared include raw BM25 over markdown, enriched BM25F, dense mean-pool retrieval, reproduced ColQwen2.5, and the larger visual retriever ColEmbed-3B-v2.
Key Experimental Results¶
Main Results¶
On the five English ViDoRe V3 domains, MIDR Hybrid achieves an average nDCG@10 of 0.6219, delivering a 23.0% relative improvement over the raw BM25 baseline (0.5057) and remaining highly competitive with our ColQwen2.5 reproduction (0.6300), leading on HR and Pharmaceuticals.
| System | CS | Finance | HR | Industrial | Pharma | EN-5 Avg. | Rel. Gain vs BM25 |
|---|---|---|---|---|---|---|---|
| BM25 over Markdown | 0.5774 | 0.4920 | 0.4851 | 0.4398 | 0.5341 | 0.5057 | - |
| Enriched BM25F (MIDR Sparse) | 0.6497 | 0.5453 | 0.5292 | 0.4705 | 0.6012 | 0.5592 | +10.6% |
| Dense Mean-Pool (EmbeddingGemma) | 0.7107 | 0.6030 | 0.5474 | 0.4541 | 0.6341 | 0.5898 | +16.6% |
| ColQwen2.5 (reproduced) | 0.7623 | 0.6276 | 0.6018 | 0.5200 | 0.6382 | 0.6300 | +24.6% |
| MIDR Hybrid (Ours) | 0.7170 | 0.6261 | 0.6043 | 0.5197 | 0.6424 | 0.6219 | +23.0% |
On the two French-document domains with English queries, generating English enrichments at index time bridges the vocabulary gap completely, outperforming ColQwen2.5 on both domains:
| System | Energy | Physics | FR-2 Avg. | Rel. Gain vs BM25 |
|---|---|---|---|---|
| BM25 over Markdown | 0.1577 | 0.1488 | 0.1532 | - |
| Enriched BM25F | 0.5132 | 0.4081 | 0.4606 | +200.7% |
| Dense Mean-Pool | 0.6042 | 0.4341 | 0.5192 | +238.9% |
| ColQwen2.5 | 0.5967 | 0.4663 | 0.5315 | +246.9% |
| MIDR Hybrid (Ours) | 0.6192 | 0.4704 | 0.5448 | +255.6% |
Ablation Study & Serving Efficiency¶
Ablation experiments isolate the contribution of specific enrichment fields and quantify the serving efficiency advantages over visual late-interaction models:
| Configuration / Ablation (English-5) | BM25F | Dense | Hybrid | Notes & Mechanism Impact |
|---|---|---|---|---|
| Markdown Only (Baseline) | 0.5057 | 0.5177 | 0.5737 | Raw text only; no visual enrichment |
| Semantic Only (Tags + Keyphrases) | 0.5057 | 0.5472 | 0.5843 | Broad conceptual matching without specific facts |
| No QA (Remove all QA pairs) | - | 0.5606 | 0.5788 | Severe regression toward the markdown baseline |
| QA Only (QA pairs + text only) | 0.5602 | 0.5622 | 0.6200 | Recovers 94% of the full schema's hybrid gain |
| Full MIDR (Role-based weights) | 0.5592 | 0.5898 | 0.6231 | Best overall coverage and multi-domain robustness |
| Retrieval System | Index Memory (MB/page) | Rel. Memory | Rel. Query Latency | ViDoRe EN-5 nDCG@10 |
|---|---|---|---|---|
| BM25 (Markdown) | ~0.005 MB | 1.0× | 1.0× | 0.5057 |
| Enriched BM25F | ~0.012 MB | 2.5× | 3.4× | 0.5592 |
| MIDR Hybrid (Ours) | 0.038 MB | 7.5× | 14.0× | 0.6219 |
| ColQwen2.5 | 0.37 MB | 65.0× | 27.9× | 0.6300 |
| ColEmbed-3B-v2 | 11.07 MB | 1940×+ | - | 0.6730 |
Key Findings¶
- Synthetic QA pairs form the core query–page bridge: Group ablations demonstrate that the QA-only variant alone scores 0.6200 nDCG@10, capturing nearly the entire gain over markdown retrieval. Conversely, removing QA drops hybrid performance to 0.5788. Leave-one-out analysis confirms that coarse QA dominates numerical queries (-0.040 when removed) while fine QA is essential for extractive queries (-0.030).
- Table summaries offer concentrated, high-impact defense against OCR flattening: Although table summaries show a modest aggregate impact across the entire corpus (-0.004 nDCG@10), their effect is intensely concentrated on table-heavy pages, where removing them causes a steep drop of 0.028 nDCG@10.
- Index-time translation acts as a cost-free cross-lingual bridge: In the French-document setup, English queries share virtually no lexical overlap with French OCR text, dropping raw BM25 to a floor of 0.1532. By materializing English enrichments at ingestion time, MIDR lifts BM25F to 0.4606 and Hybrid to 0.5448, beating ColQwen2.5 (0.5315) without query-side translation.
- Strong complementarity with visual retrieval (0.7042 Oracle): Across 1,489 English queries, roughly half show decisive winners (>0.1 nDCG@10 gap), split nearly evenly: MIDR wins 351 queries and ColQwen2.5 wins 362. MIDR excels when exact financial or pharmaceutical numbers are verbalized into QA pairs; ColQwen2.5 dominates on dense mathematical formulas, code indentation, and visual diagrams. An oracle selecting the best system per query achieves 0.7042 nDCG@10 (+0.081 over MIDR alone), demonstrating that the two paradigms encode complementary rather than redundant information.
Highlights & Insights¶
- Temporal Amortization of Multimodal Reasoning: Shifting MLLM computation from query time to index time amortizes the heavy multimodal cost over all future queries, converting recurring GPU-heavy serving bottlenecks into a one-time offline ingestion expense.
- Surgical Fact Verification Protocol: The extract–verify–refine loop avoids brittle end-to-end regeneration by auditing drafts against a five-point checklist and invoking targeted refinements only on flagged fields, preserving verified content and guaranteeing high factual fidelity.
- Drastic Index Footprint Reduction: By substituting patch-level visual tokens with structured text fields, MIDR slashes index memory by ~9× compared to ColQwen2.5 (0.038 MB vs 0.37 MB per page) and by ~290× compared to ColEmbed-3B-v2 (11.07 MB), enabling seamless deployment on conventional Elasticsearch or OpenSearch infrastructure.
Limitations & Future Work¶
- Non-Trivial Offline Ingestion Cost: The ingestion pipeline consumes an average of 2.1 MLLM calls and ~8,000 tokens per page (~23.2 seconds serial wall-clock time), creating a substantial one-time computational barrier when indexing millions of legacy document archives.
- Lag on Fine-Grained Symbolic and Code Layouts: The remaining gap to visual models is heavily concentrated in computer science (0.7170 vs 0.7623), where visual multi-vector representations better capture formulaic notation, structural diagrams, and code block formatting.
- Chart Summary Noise and Overmatching: Chart descriptions yielded a slightly negative aggregate impact (+0.003 when removed), as generic trend descriptions ("upward trajectory") frequently over-matched unrelated boolean or short queries, highlighting the need for stricter, value-anchored chart extraction prompts.
Related Work & Insights¶
- vs ColPali / ColQwen2.5: Pioneering visual late-interaction models that retain visual patch tokens for MaxSim scoring. While highly effective on diagrams and equations, they require 9× larger memory and 2× higher query latency. MIDR demonstrates that an enriched text index matches their accuracy across four of five English domains and outperforms them on cross-lingual tasks.
- vs doc2query / Doc2Query++: Traditional text expansion techniques that predict synthetic search queries from raw text. MIDR extends document expansion into the multimodal domain, grounding generation in rendered page images to recover visual structure lost by OCR.
- vs PREMIR / MLDocRAG: Prior works exploring multimodal question generation at ingestion time. MIDR distinguishes itself through a structured, multi-field schema with typed retrieval roles and an automated extract–verify–refine verification loop that audits and repairs hallucinated content before indexing.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Reframes multimodal document retrieval as a systems architecture question, demonstrating that multimodal reasoning can be shifted entirely to offline ingestion.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 7-domain evaluation covering 2,099 queries, complete field and group ablations, LOO analysis, memory/latency benchmarks, MLLM sensitivity, and oracle complementarity.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined problem formulation, precise terminology, and transparent reporting of both gains and limitations.
- Value: ⭐⭐⭐⭐⭐ Immediate practical utility for high-throughput enterprise RAG, agentic multi-hop retrieval, and resource-constrained production search engines.