Skip to content

RetrievalRouter: Joint Modality and Architecture Selection for Document Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/emrekuruu/retrieval-router
Area: LLM Reasoning
Keywords: Document Retrieval, Adaptive Routing, Modality Selection, Late Interaction, Accuracy-Latency Frontier

TL;DR

Addressing the fundamental tradeoff where multimodal late-interaction retrieval is prohibitively slow while lightweight dense retrieval fails on visually complex layouts, RetrievalRouter introduces a query-aware policy that jointly predicts both pipeline modality and architecture from query text alone, delivering 2.5% higher nDCG@5 than the strongest static pipeline while accelerating inference by 12.4× across 11 benchmarks.

Background & Motivation

Document retrieval increasingly serves as the foundational backbone for high-stakes information access in healthcare, finance, and legal compliance. Modern retrieval pipelines vary across two orthogonal architectural axes: input modality (text-based extraction via OCR versus vision-based retrieval directly over rendered page images) and matching architecture (single-vector dense bi-encoders versus token-level or patch-level late-interaction models). These structural choices impose an uncompromising trade-off: state-of-the-art multimodal late-interaction systems such as ColPali achieve unprecedented retrieval quality over visually dense layouts but exhibit latency measured in seconds per query, while lightweight text-dense or lexical retrievers (e.g., BM25) operate within milliseconds but fail silently when evidence is embedded in color charts, complex table structures, or cross-page visual hierarchies.

In production environments, engineering teams are typically forced to make a static, one-time commitment to a single retrieval configuration across the entire query workload. However, extensive empirical analysis across 11 diverse benchmarks spanning scientific papers and financial disclosures reveals that failure modes across pipelines are strictly query-dependent and profoundly asymmetric. Along the modality axis, text-only pipelines experience severe performance collapse when faced with visually structured documents; conversely, vision-language retrieval models degrade on queries demanding complex, long-range sequential textual reasoning, falling behind text retrievers due to visual patch fragmentation. Along the architecture axis, dense retrievers match the effectiveness of late interaction on the vast majority of straightforward queries at a tiny fraction of the computational expense, with late interaction proving indispensable only for a specific subset of fine-grained, compositional queries.

The root bottleneck in document retrieval is therefore not an intrinsic capacity ceiling of individual neural architectures, but rather an allocation problem stemming from static over-provisioning or under-provisioning. The paper's key insight is that optimal retrieval can be reframed as an adaptive, per-query policy routing problem across a unified multi-modal, multi-architecture action space. Core idea: formulate the joint space of pipeline modality (text vs. multimodal) and matching architecture (dense vs. late-interaction/rerank vs. lexical) as a five-arm discrete policy space, predicting the optimal configuration solely from query text via a lightweight LoRA-augmented router trained on continuous temperature-scaled soft reward targets.

Method

Overall Architecture

RetrievalRouter resolves the retrieval dilemma by dynamically directing each input query \(q\) to the lowest-cost pipeline capable of answering it, preserving heavy multimodal late-interaction passes exclusively for queries where lighter mechanisms genuinely fail. The paper evaluates seven total pipelines \(\mathcal{P} = \{\text{BM25}, \text{TD}, \text{TL}, \text{TR}, \text{MD}, \text{ML}, \text{MR}\}\). To eliminate redundant actions, the router action space is consolidated into five distinct arms \(\mathcal{A} = \{\text{BM25}, \text{TD}, \text{TR}, \text{MD}, \text{MR}\}\), while pure late-interaction baselines (TL and ML) are retained as upper-bound comparative standards.

At query time, the input string \(q\) is passed through a frozen lightweight transformer backbone with trainable low-rank adapters and mapped via a linear projection head into a probability distribution over the five routing actions. With only 15 ms of policy evaluation overhead on an NVIDIA H100 GPU, the system executes the single predicted pipeline, as depicted in the following schematic:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Text q"] --> B["Unified Modality & Architecture Action Space<br/>BM25·TD·TR·MD·MR"]
    B --> C["Lightweight LoRA Query Encoder<br/>Qwen3-0.6B + Mean-Pooling -> 1024d Vector"]
    C --> D["Single-Layer Linear Decision Head<br/>Logits s -> Softmax Distribution"]
    D --> E["Soft-Target Temperature Optimization<br/>Continuous Reward Vector & KL Distillation"]
    E --> F["Single-Parameter Pareto Tradeoff Tuning<br/>Lambda Scaling Across Accuracy-Latency Plane"]
    F -->|Dispatch Execution| G["Physical Pipeline Execution<br/>Lexical Match / Dense Vector / Multimodal Rerank"]

Key Designs

1. Unified Modality & Architecture Action Space: Overcoming Hierarchical Routing Blind Spots

Prior adaptive retrieval literature predominantly investigates binary choices within a single modality (e.g., lexical versus dense text retrieval) or routing across domain-specific expert adapters. Splitting the decision hierarchically into a "modality router" followed by an "architecture router" introduces sub-optimal cascading errors, as the decision signals for both dimensions are deeply entangled in the latent semantics of the query text. RetrievalRouter explicitly formalizes a flat, five-arm joint action space \(\mathcal{A} = \{\text{BM25}, \text{TD}, \text{TR}, \text{MD}, \text{MR}\}\). Here, BM25 handles fast keyword lookups; Text-Dense (TD) leverages Linq-Embed-Mistral for single-vector passage matching; Text-Rerank (TR) retrieves candidates via TD and rescores the Top-100 via GTE-ModernColBERT late interaction; Multimodal-Dense (MD) maps full-page screenshot images using Nomic-Embed-Multimodal into a unified vector space; and Multimodal-Rerank (MR) applies ColNomic-Embed-Multimodal patch-level late interaction over the MD candidate set. This unified action space allows the router to recognize distinct query signatures—such as color references demanding visual grounding or complex terminology requiring exact lexical matching—in a single forward pass.

2. Lightweight LoRA Query Encoder: Sub-20ms Intent Classification Architecture

To prevent policy evaluation from negating the latency gains achieved by skipping expensive retrieval pipelines, the query encoder \(E_\phi\) is built upon Qwen3-0.6B-Base. Low-Rank Adaptation (LoRA) modules are applied to attention and feed-forward projections with rank \(r=16, \alpha=32\), and dropout 0.05, introducing approximately 4 million trainable parameters while freezing the remaining 600M base model weights. Input queries are tokenized to a maximum length of 128 tokens, and the final layer hidden states are mean-pooled into a 1024-dimensional dense query vector \(\mathbf{z}_q\). The decision head \(H_\psi\) consists of a single linear layer mapping \(\mathbf{z}_q\) to routing logits \(\mathbf{s} \in \mathbb{R}^{|\mathcal{A}|}\). On an NVIDIA H100 GPU, policy inference consumes approximately 15 ms, representing only 3.8% of Text-Dense execution latency and under 0.2% of Multimodal-Late execution latency, ensuring that routing overhead remains negligible across all operational regimes.

3. Soft-Target Temperature Optimization: Resolving Evaluation Tie Discontinuities

Training a query-level retrieval router on hard categorical labels (i.e., assigning each query exclusively to a single "winner" pipeline) introduces severe label noise. In practice, multiple retrieval pipelines frequently return identical top document sets, resulting in exact ties on nDCG@5. Forcing a discrete classifier to arbitrarily choose between equivalent pipelines destabilizes training gradients. RetrievalRouter resolves this by constructing a continuous per-query reward vector over all candidate arms:

\[r_i(q) = (1 - \lambda) \cdot s_i(q) + \lambda \cdot \left(1 - \frac{L_i(q)}{\sum_{p_j \in \mathcal{A}} L_j(q)}\right)\]

where \(s_i(q) \in [0, 1]\) represents the nDCG@5 achieved by pipeline \(p_i\) on query \(q\), and \(L_i(q)\) denotes the measured wall-clock execution latency. The multi-dimensional reward vector \(\mathbf{r}_{\mathcal{A}}(q)\) is transformed into a soft target probability distribution via temperature-scaled softmax: \(\tilde{p}_i(q) = \frac{\exp(r_i(q)/\tau)}{\sum_{p_j \in \mathcal{A}} \exp(r_j(q)/\tau)}\) with \(\tau = 0.1\). While standard softmax (\(\tau=1\)) collapses the training signal into near-uniformity due to closely clustered rewards, the small temperature parameter sharply amplifies subtle efficiency and effectiveness distinctions while mathematically preserving exact ties without injecting artificial bias.

4. Single-Parameter Pareto Tradeoff Tuning: Exposing the Full Operating Continuum

Different production scenarios mandate distinct latency and throughput constraints. RetrievalRouter encapsulates the entire accuracy-efficiency frontier into a single hyperparameter \(\lambda \in [0, 1]\). Setting \(\lambda = 0\) configures an accuracy-only objective where latency is ignored, allowing the router to activate heavy reranking pipelines whenever retrieval gains are justifiable. As \(\lambda \to 1\), the loss formulation smoothly re-weights target distributions toward lower-latency alternatives (MD and BM25), converging at \(\lambda = 1\) to 100% BM25 selection. Crucially, the router does not treat cheaper pipelines merely as fallback mechanisms: even under the fully accuracy-driven \(\lambda = 0\) regime, RetrievalRouter routes 6.0% of all benchmark queries (and 20.0% of Wiki-SS queries) to BM25, demonstrating that it genuinely identifies when exact lexical matching outperforms dense representation spaces.

Loss & Training

The router is trained by minimizing the Kullback-Leibler (KL) divergence between its predicted policy distribution \(\pi_\theta(\cdot | q)\) and the soft target distribution \(\tilde{\mathbf{p}}(q)\) over the training dataset:

\[\mathcal{L}(\theta) = \sum_{q \in \mathcal{D}_{\text{train}}} D_{\text{KL}}\left(\tilde{\mathbf{p}}(q) \parallel \pi_\theta(\cdot | q)\right)\]

Queries for which all candidate pipelines achieve zero reward are masked out of the gradient at \(\lambda = 0\) to prevent uninformative gradient updates; at \(\lambda > 0\), the latency term provides a valid efficiency signal, retaining these queries in training to guide the model toward the fastest failure mode. Training is conducted on a single NVIDIA H100 80GB GPU in bfloat16 precision using AdamW (\(lr = 1 \times 10^{-4}\), weight decay 0.01, effective batch size 32 via gradient accumulation) over 2 epochs with a linear warmup across the first 10% of steps followed by cosine learning rate decay, taking approximately 20 minutes per \(\lambda\) setting.

Key Experimental Results

Main Results

RetrievalRouter was evaluated across 11 benchmark datasets spanning financial analysis, scientific publications, and open-domain retrieval (comprising over 80,000 queries), with all pipelines executed under identical hardware conditions on an NVIDIA H100 GPU. The table below presents the aggregate performance across static pipelines, prior adaptive baselines, and the proposed approach across multiple operational settings:

Retrieval System / Mode nDCG@5 MRR@5 Recall@5 Mean Latency (s) P95 Latency (s) Structural Characteristic
Static Pipelines
BM25 (Lexical) 0.510 0.476 0.613 0.019 0.046 Ultra-low latency, fails on complex visual layouts
Text-Dense (TD) 0.492 0.456 0.601 0.455 0.845 Linq-Embed-Mistral passage bi-encoder
Text-Late (TL) 0.597 0.560 0.706 2.796 5.717 GTE-ModernColBERT token-level interaction
Text-Rerank (TR) 0.604 0.572 0.702 0.924 1.665 TD bi-encoder + top-100 ColBERT rescoring
Multimodal-Dense (MD) 0.666 0.629 0.779 0.385 0.792 Nomic page screenshot single-vector retrieval
Multimodal-Late (ML) 0.737 0.704 0.834 8.283 17.744 ColPali patch late-interaction; highest static nDCG
Multimodal-Rerank (MR) 0.733 0.701 0.830 1.121 1.967 MD + top-100 ColNomic; industry deployment standard
Adaptive Strategies
Arabzadeh et al. (\(\lambda=0.0\)) 0.712 0.678 0.803 0.522 - Hard-label cost-ordered escalation heuristic
Arabzadeh et al. (\(\lambda=0.1\)) 0.715 0.681 0.806 0.476 - Budgeted strategy selection baseline
RetrievalRouter (\(\lambda=0.0\)) 0.755 0.722 0.849 0.821 1.583 Quality-maximizing operating point; dominates ML
RetrievalRouter (\(\lambda=0.1\)) 0.755 0.721 0.848 0.666 1.340 Pareto Knee: +2.5% nDCG vs ML at 12.4× speedup
RetrievalRouter (\(\lambda=0.3\)) 0.742 0.708 0.836 0.469 0.985 Dominates MR in quality while reducing latency by 58%
RetrievalRouter (\(\lambda=0.5\)) 0.707 0.672 0.805 0.314 0.721 Faster than MD (0.385s) while gaining 6.2% nDCG
RetrievalRouter (\(\lambda=0.7\)) 0.630 0.595 0.730 0.148 0.352 Efficiency-biased; matches low-latency budgets
RetrievalRouter (\(\lambda=1.0\)) 0.510 0.476 0.613 0.034 0.065 Full convergence to BM25 (+15ms policy inference)
Theoretical Bound
Per-Query Oracle (\(\lambda=0.1\)) 0.901 0.874 0.938 0.340 - Full 7-pipeline oracle upper bound

Granular Dataset Effectiveness and Systems Footprint

To illustrate how the router adapts to diverse modality characteristics and to quantify operational storage trade-offs, the following table details dataset-specific nDCG@5 metrics (\(\lambda=0\) setting) alongside the physical vector index footprints:

Dataset / System Property BM25 Text-Dense (TD) Text-Late (TL) Multimodal-Dense (MD) Multimodal-Late (ML) Ours (RetrievalRouter) Best Static Pipeline
FinReport (Financial Filing, Mixed) 0.456 0.527 0.726 0.621 0.803 0.798 0.807 (MR)
FinSlides (Slide Decks, High Visual Density) 0.183 0.410 0.387 0.799 0.902 0.886 0.903 (MR)
Wiki-SS (Wikipedia, Complex Textual) 0.745 0.702 0.784 0.648 0.743 0.785 0.784 (TL)
MP-DocVQA (Document VQA) 0.441 0.423 0.604 0.728 0.724 0.781 0.728 (MD)
TAT-DQA (Hybrid Text & Large Tables) 0.361 0.331 0.446 0.610 0.652 0.678 0.656 (MR)
Vector Index Disk Size (MB) - 3,003 (1.0×) 8,308 (2.8×) 2,618 (0.9×) 39,023 (13.0×) 52,952 (17.6× Total) -
Measured Mean Latency (s) 0.019 0.455 2.796 0.385 8.283 0.666 (\(\lambda=0.1\)) 1.121 (MR)

Key Findings

  • Simultaneous Superiority Over All Static Neural Pipelines: Operating at the Pareto knee (\(\lambda = 0.1\)), RetrievalRouter achieves 0.755 nDCG@5 at 0.666s latency. Compared with the strongest static baseline ML (0.737 nDCG@5, 8.283s), it achieves a 2.5% relative accuracy gain while operating 12.4× faster. Compared with the deployment-standard reranking pipeline MR (0.733 nDCG@5, 1.121s), it improves quality by 3.0% while running 1.7× faster (all improvements statistically significant at \(p < 0.001\)).
  • Dominance Over Prior Adaptive Strategy Selection: In accuracy-oriented settings (\(\lambda \in [0.0, 0.5]\)), RetrievalRouter outperforms the hard-label strategy selection framework of Arabzadeh et al. by 0.029 to 0.043 nDCG@5 (\(p < 0.001\)). The baseline commits greedily to the cheapest succeeding pipeline (frequently MD), discarding opportunities where rerankers would yield superior rankings, whereas soft-target optimization models the entire performance gradient.
  • Dynamic Mitigation of Modality Inversion: On visual-heavy benchmarks like FinSlides, BM25 collapses to 0.183 nDCG@5 while multimodal pipelines reach ~0.903; conversely, on the text-intensive Wiki-SS benchmark, text pipelines dominate (TL reaches 0.784, outperforming ML at 0.743). RetrievalRouter successfully identifies these domain attributes, routing 94% of Wiki-SS queries to text pipelines and achieving 0.785 nDCG@5, while routing >95% of FinSlides queries to multimodal pipelines.
  • Reranking Degradation Avoidance: Reranking does not consistently improve retrieval. On the 1,828 queries where the Oracle selects Multimodal-Dense, MD achieves an average reward of 0.93. Introducing a late-interaction rerank stage (MR) on those exact queries actively degrades the average reward to 0.70, demonstrating that the router effectively spares queries from harmful over-computation.

Highlights & Insights

  • Reframing Document Retrieval as Query-Level Dynamic Policy Routing: The paper dismantles the prevailing assumption that document retrieval must rely on an monolithic, one-size-fits-all model, demonstrating that query-level heterogeneity can be exploited to achieve both peak accuracy and low latency via an inexpensive 15 ms decision step.
  • Continuous Soft-Target Distillation Over Discrete Labels: By recognizing that empirical evaluation ties lead to catastrophic label noise in classification setups, the authors use a temperature-calibrated softmax over continuous reward vectors, providing a robust optimization paradigm for adaptive retrieval.
  • Lexical Matching as an Active Accuracy Contributor: Revealing that BM25 is selected for 6% of queries even when latency is completely unweighted (\(\lambda=0\)) proves that sparse term matching possesses structural retrieval advantages for exact-entity lookup that neural bi-encoders fail to replicate.

Limitations & Future Work

  • Index Storage and Hardware Memory Overhead: While reducing runtime latency, RetrievalRouter requires maintaining four neural vector indices and one lexical index simultaneously, requiring approximately 53 GB of disk space (dominated by ColPali's 39 GB multi-vector index) and 40 GB of GPU VRAM to keep models memory-resident. This design is best suited for latency-sensitive cloud deployments where hardware cost is secondary to user-facing latency.
  • Query-Only Semantic Ambiguity: Relying solely on the query text creates an intrinsic information ceiling. When a query is structurally underspecified (e.g., "summarize table 4"), the router cannot infer whether the referenced document page contains an image infographic or cleanly parseable text without looking at the document, leaving a 14-point gap relative to the document-aware Oracle. Incorporating lightweight exploratory probes or metadata inspection remains a promising avenue.
  • Expansion to Generative Reranking and Downstream LLM Sizing: The action space currently targets bi-encoder and late-interaction paradigms; extending dynamic routing to govern the invocation of generative listwise LLM rerankers or dynamic LLM generator sizing presents an exciting direction for compound AI architectures.
  • vs. Arabzadeh et al. (CIKM 2021) / LiteGator (2025): Previous retrieval routers operated strictly within the text domain (sparse versus dense) using discrete heuristic thresholds; RetrievalRouter generalizes this concept to the joint cross-modality (text vs. vision) and cross-architecture (dense vs. late interaction) continuum using soft-target knowledge distillation.
  • vs. ColPali (Faysse et al., 2024) / VisRAG (Ma et al., 2024): State-of-the-art visual document retrievers establish new performance upper bounds but suffer from crippling deployment latencies; RetrievalRouter proves that their benefits can be enjoyed in high-throughput production by invoking them selectively on the ~20-30% of queries that truly require visual token interaction.
  • vs. Adaptive RAG Frameworks (Self-RAG / MBA-RAG): While adaptive RAG systems dynamically govern retrieval frequency and generation verification, they treat the underlying retriever as a fixed monolith; RetrievalRouter operates beneath them, offering a dynamic retrieval primitive that adaptively scales compute per retrieval call.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First framework to formalize joint modality and architecture retrieval routing with soft-target utility distillation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 11 diverse datasets, complete physical latency audits on modern hardware, and comprehensive ablation of the Pareto frontier.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exemplary prose, transparent motivation, clean mathematical formulation, and exceptionally informative visualization designs.
  • Value: ⭐⭐⭐⭐⭐ Provides an immediately deployable and highly impactful blueprint for production-grade multimodal RAG architectures.