Skip to content

E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Vmyosotis/E2E-VisRAG
Area: Multimodal VLM
Keywords: Visual Retrieval-Augmented Generation, End-to-End Joint Training, Differentiable Top-K Selection, Gated Residual Fusion, Document Visual Question Answering

TL;DR

E2E-VisRAG discards OCR text serialization and global embedding compression by performing token-level MaxSim late interaction directly on native visual patches, introducing a sequential differentiable Top-K selector via the Straight-Through Estimator to backpropagate generation gradients into the retriever, and injecting retrieval evidence through gated residual fusion to achieve unified end-to-end optimization.

Background & Motivation

Retrieval-Augmented Generation (RAG) grounds large vision-language and language models in external knowledge, yet conventional document systems fail when confronted with visually rich documents that convey essential semantics through complex layouts, infographics, charts, and diagrams. Traditional OCR-based text pipelines flatten document images into sequential plain text, destroying two-dimensional geometric arrangements and losing non-textual figures entirely; simultaneously, OCR parsing latency and transcription errors propagate uncontrollably through downstream components. Even augmenting extracted text with VLM-generated image captions fails to preserve fine-grained localized facts, such as small coordinate labels or dense table cell values.

Although vision-language models have advanced native visual comprehension, current visual retrieval paradigms suffer from severe structural limitations. Global-embedding visual retrievers such as CLIP or SigLIP compress an entire high-resolution page into a single vector via global average pooling, obliterating localized evidence located within a specific table cell, chart legend, or handwritten note. Conversely, patch-level late-interaction visual retrievers like ColPali retain localized representations, but remain fundamentally decoupled from generation: because standard Top-K selection is a discrete, non-differentiable sampling operation, the downstream generation loss cannot backpropagate gradients into the retrieval module. As a result, the retriever is trained strictly on proxy relevance labels rather than aligned with final answer-generation needs.

Resolving the tension between fine-grained visual preservation, end-to-end gradient continuity, and robust generation against retrieval noise demands a paradigm shift from cascaded pipelines to a unified differentiable framework. Core idea: perform fine-grained token-level MaxSim late interaction directly over native visual patch embeddings, introduce a sequential differentiable Top-K selector via the Straight-Through Estimator to bridge the retrieval-generation gradient gap, and dynamically regulate evidence injection through a gated residual fusion module.

Method

Overall Architecture

The E2E-VisRAG architecture comprises three coordinated stages: Native Visual Encoding (Stage 1), Differentiable Retrieval (Stage 2), and Gated Generation (Stage 3). The pipeline takes raw document images and natural language queries without any OCR parsing, keeping the vision encoder (ViT) and language backbone parameters frozen while training lightweight Retrieval LoRA, Generation LoRA, and gated fusion adapters.

During execution, document images are mapped into patch token sequences and projected to language dimension space; the Retrieval LoRA contextualizes document patches and query tokens. The Differentiable Retrieval stage computes token-level MaxSim late-interaction relevance across all candidate documents, applying an iterative masked softmax alongside the Straight-Through Estimator (STE) to produce continuous aggregation weights for the Top-K pages. In the Gated Generation stage, the retrieved representations are fused with raw visual features via layer-normalized residual projection and a learnable gate, concatenated with system prompts and query embeddings, and passed to the Generation LoRA adapter to autoregressively decode the final answer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Document Images & Text Query<br/>OCR-Free Native Patch Partition"] --> B["Native Patch-Level Visual Encoding<br/>Frozen ViT + Projector + Retrieval LoRA"]
    B --> C["MaxSim Fine-Grained Late Interaction<br/>Token-Level Inner-Product Relevance Scoring"]
    C --> D["Sequential Differentiable Top-K Selection<br/>Iterative Softmax Masking + Straight-Through Estimator"]
    D --> E["Gated Residual Fusion<br/>LayerNorm + Projection Transform + Learnable Gate"]
    E --> F["Joint Autoregressive Generation<br/>Concat System/Query Tokens + Generation LoRA"]
    F -.->|End-to-End Generation Gradients| D
    D -.->|Supervise & Refine Retrieval Space| B

Key Designs

1. Native Patch-Level Visual Encoding: Preserving Dense Spatial Semantics without OCR Cascading Errors To eliminate the information loss and cascading transcription errors caused by OCR text flattening, as well as the loss of fine details in global average pooling, E2E-VisRAG processes raw document pages \(I_i\) directly through a frozen Vision Transformer (ViT). A multimodal projector maps the visual patch tokens into the language embedding space, yielding initial visual embeddings \(H_{\text{orig},i} \in \mathbb{R}^{S \times D}\), where \(S\) denotes the visual token count and \(D\) is the hidden dimension. Trainable Retrieval LoRA adapters integrated into the frozen backbone contextualize \(H_{\text{orig},i}\) into retrieval-oriented patch representations \(H_{\text{raw},i} \in \mathbb{R}^{S \times D}\). The input query \(q\) is similarly embedded and contextualized through the Retrieval LoRA into query tokens \(Q = \{q_1, \dots, q_m\}\). This guarantees that raw visual evidenceโ€”including table grid structures, flowcharts, and font hierarchiesโ€”is preserved in localized patch coordinates without modality conversion penalties.

2. MaxSim Fine-Grained Late Interaction: Aligning Query Terms with Localized Visual Evidence Addressing the bottleneck of global-embedding approaches where localized evidence is obscured by page-wide pooling, the framework adopts token-level late interaction. For every query token \(q_k \in Q\), the retriever identifies the maximum inner-product similarity across all visual patch representations \(h_j\) of candidate document \(d_i\), subsequently summing these maximal scores across all \(m\) query tokens: $\(s(q, d_i) = \sum_{k=1}^m \max_{j=1}^S q_k^\top h_j\)$ This late-interaction operator enables specific textual words, numerical values, and entities in the query to match corresponding visual patch regions independently. It produces a comprehensive relevance score vector \(s = (s_1, \dots, s_N)\) across all \(N\) candidate documents in collection \(\mathcal{D}\).

3. Sequential Differentiable Top-K Selection: Bridging the Disconnect via the Straight-Through Estimator Traditional RAG architectures employ discrete argmax or hard sorting to choose the Top-K evidence passages, creating a non-differentiable step that completely severs backpropagation from generation loss to retrieval weights. E2E-VisRAG overcomes this by combining temperature-scaled soft probabilities, iterative state masking, and the Straight-Through Estimator (STE). Given scaled scores \(z = s / \tau_{\text{sel}}\), the selector executes \(K\) sequential rounds. At round \(j\), a mask vector \(M^{(j-1)}\) assigns \(-\infty\) to previously selected indices and \(0\) to remaining candidates, producing selection distribution \(p^{(j)} = \text{softmax}(z + M^{(j-1)})\). The forward pass deterministically selects the discrete winner \(i^* = \arg\max_i p_i^{(j)}\) to yield one-hot vector \(K_{\text{hard}}^{(j)}\), while the backward pass computes STE aggregation weights: $\(W^{(j)} = K^{(j)}_{\text{hard}} + p^{(j)} - \text{sg}(p^{(j)})\)$ where \(\text{sg}(\cdot)\) represents the stop-gradient operator. The retrieved document representation is aggregated as \(H_{\text{ret}}^{(j)} = \sum_{i=1}^N W_i^{(j)} H_{\text{raw},i}\). Forward evaluation retains clean, discrete single-document selections, while backward backpropagation routes generation loss gradients through soft distribution \(p^{(j)}\) directly into the Retrieval LoRA parameters, aligning the retrieval embedding geometry with final question answering.

4. Gated Residual Fusion: Controlled Evidence Injection and Noise Suppression Directly concatenating retrieved patch features into the generator exposes the language model to uncalibrated retrieval noise, causing optimization instability and training divergence (DNC). Gated Residual Fusion (GRF) implements a modulated residual bridge between raw and retrieval-adapted visual spaces. For each retrieved document \(d_{r_k}\), the module applies independent layer normalization to the raw ViT feature \(H_{\text{orig},i}\) and retrieval-contextualized feature \(H_{\text{raw},i}\), computing the residual increment \(\Delta H_i = \tilde{H}_{\text{raw},i} - \tilde{H}_{\text{orig},i}\). This difference is projected through a learnable weight matrix \(W_p \in \mathbb{R}^{D \times D}\), and merged into the generation input via a learnable gate vector \(\gamma \in \mathbb{R}^D\): $\(H_{\text{input},i} = H_{\text{orig},i} + \sigma(\gamma) \cdot (W_p \Delta H_i)\)$ The gate is initialized to \(\sigma(\gamma) \approx 0.2\) (\(\gamma = -1.3863 + \mathcal{N}(0, 0.02)\)). In early training epochs, the model relies predominantly on stable raw visual semantics, progressively learning to absorb retrieval-specialized features while actively filtering out irrelevant page artifacts.

Loss & Training

The framework is optimized in two progressive stages under frozen backbone parameters: - Stage 1 (Retrieval Pre-Warming): The vision encoder, multimodal projector, and language backbone remain frozen. Only Retrieval LoRA adapters are trained using the InfoNCE contrastive objective: $\(\mathcal{L}_{\text{retrieval}} = -\log \frac{\exp(s(q, d^+) / \tau_{\text{cl}})}{\sum_{d' \in \mathcal{B}} \exp(s(q, d') / \tau_{\text{cl}})}\)$ where \(d^+\) is the ground-truth document, \(\mathcal{B}\) denotes the batch containing in-batch negative documents, and \(\tau_{\text{cl}} = 0.07\). - Stage 2 (End-to-End Joint Fine-Tuning): All trainable componentsโ€”Retrieval LoRA, Generation LoRA, and Gated Residual Fusion (\(W_p, \gamma\))โ€”are trained jointly under a combined multi-task objective: $\(\mathcal{L} = \lambda_r \mathcal{L}_{\text{retrieval}} + \lambda_g \mathcal{L}_{\text{generation}}\)$ with \(\lambda_r = 0.1, \lambda_g = 1.0\). Generation loss \(\mathcal{L}_{\text{generation}} = -\sum_t \log P(y_t \mid y_{<t}, H_{\text{final}})\) passes through the differentiable Top-K selector, directly supervising the retrieval representation space with generation-aware gradients.

Key Experimental Results

Main Results

Evaluation is conducted on two multimodal document benchmarks: Vidore (500 validation QA pairs across Infographics, ArXiv, DocVQA, TAT-DQA, and PDF domains) and MMLongBench-Doc (838 answerable questions across 135 documents, averaging 48.3 pages per document). Generators utilize Qwen2.5-VL-3B-Instruct (and 7B for reference baselines). Correctness is assessed using GPT-4o-mini as an automated LLM-as-Judge over 5 independent runs.

Method Generator Vidore R@1 (%) Vidore R@5 (%) Vidore Acc@1 (%) Vidore Acc@5 (%) MMDoc R@1 (%) MMDoc R@5 (%) MMDoc Acc@1 (%) MMDoc Acc@5 (%)
OCR + BM25 3B 31.00 44.60 25.30 29.20 21.20 44.00 11.20 14.40
OCR + BGE 3B 60.20 71.40 38.40 39.10 35.20 62.80 15.40 21.10
OCR + Cap + BM25 3B 47.40 62.60 32.80 35.60 31.20 57.50 13.70 18.20
OCR + Cap + BGE 3B 62.60 75.80 43.00 46.00 40.00 66.60 19.80 23.40
SigLIP 3B 25.80 39.20 22.50 29.80 24.30 53.40 15.30 23.10
ColPali 3B 78.80 90.60 58.40 58.90 45.30 73.70 24.80 30.10
E2E-VisRAG (Ours) 3B 83.60 94.40 61.60 62.44 46.10 74.10 26.63 30.79

Ablation Study

The contribution of each architectural innovation is quantified through ablations on Vidore and MMLongBench-Doc:

Configuration Vidore R@1 (%) Vidore R@5 (%) Vidore Acc@1 (%) Vidore Acc@5 (%) MMDoc R@1 (%) MMDoc R@5 (%) MMDoc Acc@1 (%) MMDoc Acc@5 (%) Notes
Full Model 83.60 94.40 61.60 62.44 46.06 74.10 26.63 30.79 Full E2E-VisRAG framework
w/o E2E 82.60 93.80 60.84 60.32 45.53 74.08 26.21 29.16 Disables Stage 2 joint training; decoupled pipeline
w/o Diff. Top-K 83.20 93.80 61.40 61.20 45.58 73.80 26.01 29.12 Replaces STE with standard hard Top-K, blocking gradients
w/o GRF NA NA NA NA NA NA NA NA Direct concatenation of retrieved features; training diverges (DNC)

Under the multi-page split of MMLongBench-Doc, E2E-VisRAG demonstrates remarkable superiority: on multi-page questions, E2E-VisRAG achieves 19.62% Acc@1 and 18.28% Acc@5, substantially outperforming ColPali3B (11.28% Acc@1 / 17.44% Acc@5) and OCR+BGE3B (6.17% Acc@1 / 11.89% Acc@5), highlighting its resilience in cross-page visual reasoning.

Key Findings

  • Generation gradients directly sharpen retrieval quality: Eliminating joint optimization (w/o E2E) reduces Vidore Acc@5 by 2.12 points (62.44% \(\to\) 60.32%) and drops standalone retrieval Recall@1 from 83.60% to 82.60% (with DocVQA gaining +3.1% under E2E). This confirms that generation loss feedback actively restructures the retrieval representation space beyond contrastive pre-training limits.
  • Differentiable Top-K is essential for pipeline synergy: Replacing the STE selector with hard Top-K (w/o Diff. Top-K) cuts MMLongBench-Doc Acc@5 from 30.79% to 29.12% and degrades Recall@1, demonstrating that without gradient flow, evidence selection remains trapped in sub-optimal local minima.
  • Gated Residual Fusion prevents representation collapse: Removing GRF leads immediately to training divergence (DNC), proving that unmodulated raw feature concatenation introduces unbearable feature noise into the generation backbone.
  • Superior latency-accuracy efficiency: While SigLIP is fastest (1,533 ms/q), its Acc@5 is a dismal 31.20%. E2E-VisRAG achieves 62.44% Acc@5 in 1,562 ms/q total latencyโ€”over twice as fast as OCR+BGE (3,420 ms/q), where OCR parsing alone takes 2,491 ms/pg.

Highlights & Insights

  • Straight-Through Estimator unlocks continuous end-to-end RAG: Using STE to perform hard selection in forward inference and soft-probability backpropagation in the backward pass elegantly circumvents discrete sampling bottlenecks in visual document pipelines.
  • Decisive shift away from OCR preprocessing: In visual benchmarks like DocVQA, patch-level matching achieves 67.70% R@1 versus only 14.91% for OCR+BM25, proving that native visual patches preserve geometric layouts that OCR irrevocably destroys.
  • Safe feature injection via calibrated residual gates: Initializing the fusion gate at \(\sigma(\gamma) \approx 0.2\) provides a gentle initialization path, enabling the model to retain visual grounding before absorbing specialized retrieval cues.

Limitations & Future Work

  • Linear memory scaling over in-batch candidate pages: Differentiable Top-K selection retains computation graphs for all in-batch candidates, scaling with \(\mathcal{O}(N)\) memory complexity. Scaling to millions of external corpus documents will require hierarchical clustering or memory bank mechanisms.
  • Visual token budget computation overhead: Expanding visual patch limits from 784 to 1024 tokens delivers noticeable accuracy gains (Vidore Acc@1 rises to 64.40%), but significantly increases MaxSim computation and attention memory.
  • Evaluation restricted to English corpora: Benchmarks evaluate only English documents. Transferability to non-Latin scripts, low-resource languages, and distorted scans remains open for future research.
  • vs OCR-Based RAG (BM25 / BGE / Captioning): OCR pipelines discard visual typography, incur high computational latency (2,491 ms/pg), and introduce cascading recognition errors. E2E-VisRAG is fully native and OCR-free, doubling accuracy while halving latency.
  • vs Global-Embedding Visual Retrieval (SigLIP): SigLIP pools whole pages into single vectors, collapsing granular table and diagram details (scoring only 25.80% R@1 on Vidore). E2E-VisRAG preserves fine details via patch-level MaxSim late interaction.
  • vs Decoupled Patch Retrieval (ColPali): While ColPali established late-interaction patch retrieval, its retriever is decoupled from generator supervision. E2E-VisRAG connects both stages with a differentiable gradient bridge, lifting Vidore Recall@1 from 78.80% to 83.60% and Acc@5 from 58.90% to 62.44%.

Rating

  • Novelty: โญโญโญโญโญ First end-to-end visual RAG framework establishing differentiable Top-K gradient backpropagation from answer generation to native patch retrieval.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive cross-domain benchmark evaluation, single- vs. multi-page breakdowns, component-wise ablations, and detailed latency audits.
  • Writing Quality: โญโญโญโญโญ Clearly structured exposition, precise mathematical definitions of differentiable selection operators, and rigorous error analyses.
  • Value: โญโญโญโญโญ Sets a new architectural milestone for document intelligence, multimodal RAG systems, and end-to-end visual question answering.