Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Collab-Gen/Doc-REFRAG
Area: Information Retrieval/RAG
Keywords: Multimodal Document RAG, Visual Token Compression, RL Selector, Coarse-to-Fine Restoration, DocLongRAG
TL;DR¶
To resolve visual token explosion and retrieval noise in multi-image multimodal document RAG, Doc-REFRAG introduces coarse-level chunk compression paired with a lightweight RL-based selector that selectively expands question-critical regions, supported by DocLongRAG with hard-negative mining, achieving state-of-the-art accuracy with markedly reduced TTFT and TTIT.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have made tremendous strides in visual document understanding, yet their static parametric knowledge remains vulnerable to hallucination and factual drift. Augmenting MLLMs with retrieval-augmented generation (RAG) over visually rich document collections—such as scientific PDFs, presentation slides, and financial reports—provides essential external grounding. However, most existing document-aware MLLMs are trained exclusively on individual pages or short, coherent multi-page documents (e.g., Doc-750K, OpenDocVQA). When deployed in real-world RAG workflows, the top retrieved candidate documents originate from diverse, heterogeneous sources, exhibiting substantial redundancy, semantic overlap, and contradictory evidence. This severe distributional shift drastically degrades the reasoning performance of conventional MLLMs.
Orthogonal to the reasoning challenge is the computational bottleneck caused by visual token explosion. Processing dozens of high-resolution retrieved document images produces tens of thousands of visual tokens, incurring catastrophic memory consumption and prefill latency. Existing visual token compression approaches—whether training-free pruning or static projection downsampling—fundamentally rely on query-agnostic assumptions: they prioritize visually complex or text-dense patches while discarding sparse or blank areas. In realistic RAG contexts, however, visually intricate passages are frequently irrelevant distractors, while critical answers may reside in concise, sparse blocks. Static pruning permanently loses fine-grained evidence without meaningfully filtering question-irrelevant computation.
To overcome this efficiency-accuracy dilemma, multimodal document RAG requires a paradigm shift from irreversible static pruning to dynamic, question-guided restoration. Core idea: rethink multi-image visual compression as global coarse chunk embedding combined with lightweight, RL-driven selective expansion for question-critical regions, powered by the realistic noisy retrieval corpus DocLongRAG and a three-stage progressive training curriculum.
Method¶
Overall Architecture¶
Doc-REFRAG consists of four primary components: a document-specialized vision encoder built upon DocOwl2, a two-layer MLP chunk projector \(\phi\), a lightweight RL-based selector \(\pi_\theta\) (<1M parameters), and a 7B decoder-only foundation language model. The overall workflow executes a three-phase compress-select-expand pipeline: \(n\) retrieved document images are first partitioned into non-overlapping spatial chunks and projected into coarse embeddings; the lightweight selector evaluates the question against all chunk embeddings in a single forward pass, predicting binary expansion decisions under an explicit token budget; critical chunks are expanded back into their original constituent fine-grained visual tokens, while non-critical chunks remain compact embeddings, forming a hybrid sequence fed into the language model.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Retrieved Document Images (n images) + Question q"] --> B["Stage 1: 2D Raster Chunk Coarse Compression<br/>Each image with 324 tokens partitioned into L chunks (k=3) via MLP"]
B --> C["Stage 2: Lightweight RL Selector Dynamic Budget Decision<br/>2-layer Transformer evaluates relevance, predicts binary flag δ under budget p=0.2"]
C -->|δ=1 Selected| D["Stage 3: Question-Relevant Chunk Fine Expansion<br/>Restored to original k fine-grained visual tokens"]
C -->|δ=0 Non-Selected| E["Retain Coarse Chunk Embeddings<br/>Serve as global low-cost context"]
D --> F["Hybrid Sequence Concatenation & Answer Decoding<br/>Fed into 7B Decoder LLM for generation"]
E --> F
Key Designs¶
1. 2D Raster Chunk Coarse Compression: Preserving Spatial Topology and Semantic Compactness
To mitigate the linear token scaling across dozens of retrieved images without suffering irreversible information loss, Doc-REFRAG avoids premature pruning. The vision encoder maps each page to \(m = 324\) visual tokens, which are partitioned in raster scan order into \(L = m / k\) non-overlapping chunks \(G(i) = \{C_1^i, \dots, C_L^i\}\), with default chunk size \(k = 3\) (yielding \(L = 108\) chunks per page). Because the vision encoder aggregates local 2D spatial layouts with global self-attention, consecutive tokens in raster order naturally correspond to contiguous physical regions on the document page. A two-layer MLP projector \(\phi\) compresses the \(k\) tokens of each chunk into a single \(d\)-dimensional embedding:
This mapping reduces the token footprint to \(1/k\) while fully preserving the underlying 2D layout coordinates, providing deterministic spatial anchors for downstream selective restoration.
2. Lightweight RL Selector with Dynamic Budgeting: Task-Reward-Driven End-to-End Decision Policy
To pinpoint essential evidence from hundreds of coarse chunk embeddings, Doc-REFRAG introduces a compact policy network \(\pi_\theta\) (a two-layer Transformer encoder with hidden dimension 256, 8 attention heads, and <1M parameters). Unlike dense multi-vector retrievers such as ColQwen, which incur a prohibitive 9.3s latency by computing exhaustive patch-level similarity aggregations, the RL selector processes question tokens \(t\) and all chunk embeddings \(\{c_j^i\}\) jointly in a single forward pass within 3.7s. To prevent token explosion during decoding, the policy strictly enforces an expansion budget:
where \(p \in (0, 1]\) governs the expansion fraction (uniformly sampled from \([0.1, 0.3]\) during training for budget robustness, and set to \(p = 0.2\) at inference). Only the top \(T\) chunks with \(\delta_j^i = 1\) are expanded to their original \(k\) fine tokens, while the remaining chunks stay as coarse vectors. The selector is trained using a GRPO framework with clipped PPO updates, directly using the frozen decoder's answer accuracy as the scalar reward. Trajectories are sampled in groups of \(G = 8\), computing relative advantage \(A_i = r(i) - \frac{1}{G}\sum_{g=1}^G r(g)\), which eliminates heuristic bias toward visually salient regions.
3. Three-Stage Progressive Training Curriculum with DocLongRAG Hard Negatives: Aligning Hybrid Sequences Under Realistic Noise
Bridging coarse chunk embeddings, fine-grained visual tokens, and autoregressive language generation requires progressive adaptation. Doc-REFRAG establishes a three-stage curriculum: Stage 1 performs Single-Image Reconstruction on DocStruct4M with a frozen decoder, training the encoder alignment and MLP projector on text parsing and text localization to ensure chunk embeddings faithfully preserve text semantics; Stage 2 conducts Multi-Image Continual Pretraining (CPT) on OpenDocVQA (20% expansion), Doc-750K (20% expansion across multi-page layouts), and DocLongRAG (10% expansion with random noisy chunk activation) using LoRA (rank=64) on the 7B decoder, teaching the model to integrate hybrid token representations across extended contexts; Stage 3 freezes the decoder and trains the RL selector policy on DocLongRAG via GRPO.
Accompanying this architecture is DocLongRAG, a dedicated large-scale benchmark of 343,474 question-answer pairs over 3.3 million document images. DocLongRAG systematically filters context-independent questions, unanswerable instances, and unethical content. It introduces an automated hard-negative mining pipeline: for each ground-truth document image, a document-aware retriever identifies top-\(k\) candidates; the top-3 results (which frequently contain partial answer evidence in human validation) are excluded, and \(m \in [20, 30]\) hard negatives from rank 4 onward are interleaved with the relevant document, producing an average of 37.4 images per query. This exposure to long-range distractor context during pretraining equips the model with robust noise resilience.
Loss & Training¶
Stages 1 and 2 optimize standard cross-entropy loss over predicted text tokens. In Stage 3, the policy network \(\pi_\theta\) is updated via the clipped PPO objective with KL regularization:
where \(\rho(\delta) = \frac{\pi_\theta(\delta)}{\pi_{\text{ref}}(\delta)}\) is the importance sampling ratio, clipping threshold \(\epsilon = 0.2\), KL penalty coefficient \(\beta = 0.1\), and \(\pi_{\text{ref}}\) is initialized from SFT warm-up weights. All stages train for 2 epochs on 8 × NVIDIA RTX 3090 GPUs (24GB VRAM each), where Stage 2 accounts for the dominant computational investment while Stages 1 and 3 are exceptionally lightweight.
Key Experimental Results¶
Main Results¶
Evaluated across six standard multimodal document benchmarks (ChartQA, SlideVQA, InfoVQA, DUDE, ViDoRe-v2, and MMDocIR) with top-20 candidate images retrieved via Jina-Embeddings-V4 (Recall@20 = 93.4%), Doc-REFRAG is benchmarked against 11 representative MLLMs in accuracy, Time-To-First-Token (TTFT), and Time-To-Inter-Token (TTIT):
| Model | Size | TokenV / img | ChartQA | SlideVQA | InfoVQA | DUDE | Vidore | MMDocIR | #Acc. (%) | #TTFT (s) | #TTIT (s) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| InternVL3.5 | 8B | ~2,304 | 56.8 | 53.1 | 48.1 | 45.8 | 42.1 | 51.5 | 49.5 | 2.78 | 0.92 |
| LLaVA-OneVision | 8B | ~1,920 | 52.1 | 50.9 | 46.3 | 47.2 | 42.7 | 49.0 | 48.0 | 4.03 | 0.05 |
| MiniCPM-V-2.6 | 8B | ~576 | 54.8 | 52.3 | 48.5 | 46.4 | 41.1 | 48.8 | 48.6 | 5.80 | 0.12 |
| Qwen3-VL | 8B | ~512 | 58.2 | 59.2 | 55.7 | 52.5 | 43.5 | 52.4 | 51.3 | 5.91 | 0.05 |
| MQT-LLaVA† | 7B | 256 | 35.7 | 39.3 | 33.4 | 40.5 | 37.4 | 34.2 | 36.7 | 3.43 | 0.05 |
| EPIC† | 7B | 256 | 39.7 | 41.1 | 42.7 | 32.4 | 33.3 | 38.2 | 37.9 | 3.42 | 0.06 |
| LLaVA-Mini† | 7B | 144 | 35.4 | 38.5 | 44.6 | 37.5 | 39.1 | 40.2 | 39.2 | 2.67 | 0.06 |
| Docopilot | 8B | ~2,304 | 58.7 | 56.6 | 49.4 | 48.6 | 45.7 | 46.5 | 50.9 | 1.81 | 2.25 |
| VDocGenerator | 3B | ~1,920 | 53.3 | 42.8 | 43.2 | 37.9 | 34.6 | 36.8 | 41.4 | 9.20 | 6.40 |
| DocOwl2† | 7B | ~324 | 57.5 | 53.2 | 45.2 | 44.7 | 47.5 | 45.2 | 48.9 | 3.94 | 0.07 |
| TokenPacker† | 7B | 144 | 46.4 | 39.9 | 41.3 | 33.5 | 36.7 | 35.9 | 37.5 | 4.43 | 0.05 |
| Doc-REFRAG (Ours) | 7B | ~150 | 61.2 | 60.8 | 58.7 | 53.2 | 57.3 | 54.9 | 57.5 | 2.57 | 0.07 |
(Note: † denotes models utilizing visual token compression mechanisms.)
Ablation Study¶
Ablation investigations quantify the specific contributions of selection strategies, training stages, and chunk size configurations:
Table 1: Performance and Latency Across Selection Strategies (8×RTX 3090)
| Strategy | ChartQA | SlideVQA | InfoVQA | DUDE | Vidore | MMDocIR | Average (%) | Selection Latency (s) |
|---|---|---|---|---|---|---|---|---|
| Random Selection | 39.4 | 42.5 | 48.3 | 33.2 | 38.6 | 30.8 | 38.8 | - |
| Perplexity Heuristic | 46.5 | 48.3 | 47.2 | 39.6 | 40.4 | 40.3 | 43.7 | 20.7 |
| ColQwen Selector | 56.6 | 54.0 | 59.5 | 49.8 | 53.5 | 52.3 | 53.6 | 9.3 |
| RL Selector (Ours) | 61.2 | 60.8 | 58.7 | 53.2 | 57.3 | 54.9 | 57.5 | 3.7 |
Table 2: Ablation Study of Training Tasks (Accuracy %)
| Variant | Vidore | MMDocIR | DUDE | Average (%) | Note |
|---|---|---|---|---|---|
| Doc-REFRAG (Full) | 53.2 | 57.3 | 54.9 | 55.1 | Full three-stage curriculum |
| w/o Single-image Recon. | 20.2 | 16.5 | 22.5 | 19.7 | Severe representation collapse (-35.4%) |
| w/o Curriculum Learning | 24.5 | 22.8 | 25.4 | 24.2 | Shuffled multi-image inputs degrade to 24.2% |
| w/o Both | 7.2 | 8.5 | 8.4 | 8.0 | Complete inability to process hybrid tokens |
Key Findings¶
- Single-Image Reconstruction is Indispensable: Omitting Stage 1 text parsing and localization supervision triggers a precipitous drop in average accuracy from 55.1% to 19.7%. This establishes that an unregularized MLP projector fails to preserve readable linguistic structures when compressing visual tokens into chunk embeddings.
- RL Selection Outpaces Both Heuristics and Dense Retrievers: The learned RL selector achieves 57.5% average accuracy, heavily surpassing Random (38.8%) and Perplexity-based ranking (43.7%). Although ColQwen remains competitive on InfoVQA, its patch similarity aggregation takes 9.3s; the RL selector achieves higher overall accuracy while slashing latency by 2.5× to 3.7s.
- \(k=3\) Achieves the Sweet Spot of Compression and Fidelity: Adjusting chunk size from \(k=2\) (194 tokens/image) to \(k=3\) (150 tokens/image) actually improves average accuracy from 55.5% to 57.5%, indicating that coarse grouping suppresses high-frequency background noise. Conversely, aggressive compression to \(k=6\) (108 tokens/image) causes accuracy to drop to 52.3%, severely impairing complex reasoning in ChartQA and InfoVQA.
Highlights & Insights¶
- Question-Guided Elastic Reconstruction Transcends Static Saliency: Prior compression frameworks naively discard sparse regions and over-allocate tokens to dense text, which in RAG setups often represents irrelevant distractors. Doc-REFRAG dynamically expands only the chunks conditioned on question relevance, achieving true on-demand compute.
- Internalizing Reranking into a Sub-1M Parameter Policy: Traditional multi-stage RAG pipelines rely on heavy Cross-Encoder rerankers (such as MonoQwen2 taking ~15.4s for top-20 candidates), whereas Doc-REFRAG incorporates candidate filtering and selective visual expansion directly into a tiny 2-layer Transformer, completing end-to-end inference in 6.1s while improving accuracy from 53.2% to 57.5%.
- DocLongRAG Curriculum for Real-World Retrieval Noise: By deliberately excluding top-3 ambiguous candidates and synthesizing sequences with 20–30 interleaved hard negatives (averaging 37.4 pages), DocLongRAG trains models to withstand distractors and multi-document contradictions. Factorial analysis confirms that DocLongRAG contributes a +3.9% gain alone, and +2.3% synergistic interaction with the architecture.
Limitations & Future Work¶
- Static Chunk Granularity: The chunk size \(k=3\) is fixed prior to training and cannot dynamically adapt at test time. On document images with more than 4,000 characters, parsing ANLS experiences a ~5% drop, highlighting the need for adaptive multi-scale chunking.
- Sparse Scalar Rewards in RL Training: The policy network is optimized solely against final answer accuracy, lacking token- or chunk-level attribution. Error analysis indicates that 64% of remaining failures stem from dense multi-paragraph textual reasoning; incorporating dense process rewards or chunk-level supervision could further refine selection precision.
- Language and Noise Distribution Bias: DocLongRAG is currently restricted to English documents. CJK languages with higher information density per visual patch may favor smaller chunk dimensions (\(k=1\) or \(k=2\)). Furthermore, programmatic hard-negative synthesis may not encompass all long-tail industrial retrieval patterns.
Related Work & Insights¶
- vs DocOwl2 / TokenPacker: Conventional document foundation models apply fixed downsampling or hierarchical pooling uniformly across all pages, causing irreversible detail loss. Doc-REFRAG establishes a hybrid "coarse foundation + question-guided fine expansion" architecture that preserves high-fidelity spatial details for target evidence.
- vs SparseVILA / IVTP: Prior instruction-aware visual token pruning methods depend on attention rollout heuristics or decode-time KV cache searches without end-to-end supervision. Doc-REFRAG formulates chunk expansion as an RL policy optimized via GRPO against task accuracy, achieving higher accuracy at a fraction of the inference latency.
- vs MonoQwen2 Reranking Pipeline: Traditional pipelines incur severe latency (15.4s) and risk cascading truncation errors by reranking candidates down to top-5 before feeding them into an MLLM. Doc-REFRAG ingests all top-20 candidates under coarse chunking and selectively expands relevant regions in 6.1s, yielding a superior accuracy-efficiency frontier.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant reframing of multimodal token compression into an RL-driven, budget-constrained dynamic expansion paradigm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 benchmarks, 11 strong baselines, full system latency profiles, and factorial ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, lucid narrative progression, and rigorous ablation isolating data versus architectural gains.
- Value: ⭐⭐⭐⭐⭐ Directly addresses the critical prefill latency and token explosion barriers in real-world multimodal RAG deployment.