Skip to content

Adaptive Knowledge Routing for Complex Reasoning in LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Reasoning
Keywords: listwise reranking, soft prompt compression, retrieval-augmented generation, knowledge distillation, efficiency optimization

TL;DR

Addressing the severe latency bottleneck of listwise LLM rerankers that ingest lengthy concatenations of full-text documents, this paper presents RRK, an efficient listwise reranker operating directly over multi-token soft-compressed representations that achieves 5ร—โ€“20ร— latency reductions on a 7B backbone while matching or surpassing 0.6Bโ€“4B full-text rerankers across standard and long-document benchmarks.

Background & Motivation

Modern information retrieval (IR) and retrieval-augmented generation (RAG) pipelines fundamentally rely on a two-stage retrieval architecture: an initial coarse stage rapidly selects candidate documents, followed by an expressive reranker that refines candidate ordering via fine-grained relevance evaluation. With the rapid evolution of large language models and autonomous agent workflows that repeatedly query external toolkits and structured corpora, high-quality reranking has become the decisive gating factor for downstream performance: context precision directly dictates whether complex multi-step reasoning succeeds or collapses into hallucination. Recently, listwise LLM rerankers such as RankGPT and RankZephyr have demonstrated superior retrieval effectiveness by ingesting the entire candidate pool simultaneously, allowing the attention mechanism to model cross-document comparisons rather than evaluating each document in isolation.

However, this gain in retrieval quality comes at a prohibitive computational cost. Listwise LLM rerankers require concatenating the user query with dozens of candidate documents into a single prompt of several thousand tokens, triggering quadratic computational complexity in self-attention; for models at the 7B parameter scale or larger, this massive forward footprint renders real-time production deployment impractical. Prior efficiency-oriented solutions have attempted to circumvent this obstacle by either pruning model size down to sub-billion parameters, severely truncating input text length, or substituting candidate documents with single-vector first-stage retrieval embeddings (such as PE-Rank). Inevitably, these compromises degrade representational capacity, discarding the nuanced textual semantics required for subtle relevance discrimination.

Meanwhile, soft prompt compression frameworks developed for RAG (e.g., PISCO, ICAE) have demonstrated that extensive document contexts can be compressed into a handful of continuous virtual tokens while preserving answer fidelity in generation decoders. This breakthrough sparks a compelling research question: can soft-compressed representations replace raw textual documents directly within listwise rerankers, circumventing quadratic sequence scaling while preserving cross-candidate reasoning? The core idea is to bypass the conventional reliance on full-text prompt concatenation by introducing RRK, an efficient listwise reranker operating directly over multi-token soft-compressed document representations, which pairs an offline 8-token document compressor with a query-sandwich decoder architecture to optimize RankNet preference loss via end-to-end distillation, achieving a 5ร—โ€“20ร— speedup on a 7B backbone with competitive or superior nDCG@10 accuracy.

Method

Overall Architecture

RRK (ReRanKer with Compressed Document Representations) is composed of two jointly fine-tuned neural components: 1. Document Soft-Compressor (\(f_{\theta_c}\)): A LoRA-adapted causal transformer that maps each raw document \(d_i\) into a compact sequence of \(l\) learnable memory token embeddings \(c_i = (c^1_i, \dots, c^l_i) \in \mathbb{R}^{l \times d_{\text{model}}}\) during offline indexing; 2. Listwise Decoder Reranker (\(g_{\theta_r}\)): A LoRA-adapted causal decoder that ingests the natural language query \(q\) alongside the compressed token representations of all candidate documents, structured within a bidirectional query-sandwich sequence, to yield contextual relevance scores in a single forward pass without autoregressive token generation.

During offline processing, the corpus documents are encoded once into fixed-length compressed tokens (\(l=8\)), eliminating raw text tokenization during live query servicing. At inference time, given a query \(q\) and a candidate pool \(\mathcal{D}_k = \{d_1, \dots, d_k\}\), RRK fetches the precomputed compressed arrays and constructs the input sequence \(X = (q; c_1; \text{[SEP]}; c_2; \text{[SEP]}; \dots; c_k; \text{[SEP]}; q)\). The model extracts the contextual query representation \(q_{\text{rep}} = H_{|X|}\) from the terminal query token and the document representations \(h_i\) from each separator token \(\text{[SEP]}\), computing candidate relevance via normalized cosine similarity \(s_i = \cos(q_{\text{rep}}, h_i)\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Query q and Candidate Document Set D_k"] --> B["Memory Token Appending & Offline Multi-Token Compression<br/>Compress long document text into fixed-length latent tokens"]
    B --> C["Bidirectional Query Sandwich & Contextual Sequence Assembly<br/>Sandwich compressed candidate tokens between twin query tokens"]
    C --> D["Separator Hidden State Extraction & Single-Pass Cosine Scoring<br/>Extract contextual SEP embeddings and compute similarity with query"]
    D --> E["End-to-End Joint Distillation & RankNet Preference Optimization<br/>Backpropagate ranking gradients through decoder into compressor"]
    E --> F["Output Reordered Candidate Ranking and Relevance Scores"]

Key Designs

1. Memory Token Appending & Offline Multi-Token Compression: Overcoming Representational Bottlenecks of Single Embeddings

Existing efficient rerankers such as PE-Rank or E2Rank compress each candidate document into a solitary retrieval vector, which lacks sufficient expressive capacity to retain nuanced domain facts and conditional relevance signals. RRK addresses this limitation by adapting the PISCO soft-compression framework: for every document \(d_i\), it appends \(l=8\) trainable memory tokens \((m_1, \dots, m_l)\) to the document text, passing \((d_i; m_1, \dots, m_l)\) through the compressor to extract the final-layer hidden activations of the memory tokens as the compressed vector \(c_i = (c^s_i)_{s=1,\dots,l}\). This achieves an immediate 16ร— length compression relative to standard 128-token passages and up to a 256ร— compression factor on 2048-token documents. Because compression is executed offline, precomputed representations are indexed in storage; vector retrieval during inference accounts for less than 10% of total query turnaround, completely offloading online document parsing and raw text self-attention.

2. Bidirectional Query Sandwich & Contextual Sequence Assembly: Enabling Cross-Candidate Interaction in Causal Decoders

Standard causal language models utilize lower-triangular attention masks, preventing preceding tokens from attending to subsequent tokens in the sequence. To enable bidirectional cross-candidate visibility without modifying native FlashAttention kernels or introducing custom non-causal attention operators, RRK devises a query-sandwich formatting strategy: $\(X = \left(q; \, c_1; \, \text{[SEP]}; \, c_2; \, \text{[SEP]}; \, \dots; \, c_k; \, \text{[SEP]}; \, q\right)\)$ The leading query tokens allow causal attention to condition each compressed document block \((c_1, \dots, c_k)\) on the initial query intent while permitting downstream candidate blocks to attend to upstream candidates. Crucially, duplicating the query text \(q\) at the sequence terminus allows the trailing query tokens to attend across all candidate representations and their relational boundaries simultaneously. Whereas standard listwise rerankers ingest \(|q| + k|d|\) tokens with \(\mathcal{O}((|q| + k|d|)^2)\) computational complexity, RRK shrinks the sequence length to \(2|q| + k(l+1)\), yielding an order-of-magnitude reduction in quadratic attention operations for candidate pools of \(k=50\) documents.

3. Separator Hidden State Extraction & Single-Pass Cosine Scoring: Eliminating Autoregressive Decoding Latency

Generative listwise rerankers such as RankGPT generate ranked document identifiers sequentially, incurring substantial decoding latency and risking syntactic formatting failures. RRK eliminates autoregressive generation altogether, employing a non-generative representation-based scoring formulation. Passing the assembled sequence \(X\) through the decoder in a single forward pass, the model extracts the final hidden activation of the terminal query token as the holistic query representation: $\(q_{\text{rep}} = H_{|X|}\)$ Simultaneously, the hidden state \(h_i \in \mathbb{R}^{d_{\text{model}}}\) corresponding to the separator token \(\text{[SEP]}\) immediately following compressed block \(c_i\) is harvested as the contextualized document representation. The scalar relevance score \(s_i\) is directly computed via cosine similarity: $\(s_i = \cos(q_{\text{rep}}, \, h_i) = \frac{q_{\text{rep}} \cdot h_i}{\|q_{\text{rep}}\|_2 \|h_i\|_2}\)$ This design reduces multi-second autoregressive generation to a single tensor forward pass followed by parallel vector inner products, simultaneously unlocking high batching throughput across multiple queries on standard GPU hardware.

4. End-to-End Joint Distillation & RankNet Preference Optimization: Task-Adaptive Alignment of Compression Latents

Directly deploying a compression model pretrained exclusively on generative question answering yields suboptimal ranking results due to an objective mismatch between text reconstruction and relevance ordering. RRK couples the LoRA-adapted compressor \(f_{\theta_c}\) and decoder reranker \(g_{\theta_r}\) via end-to-end joint distillation under a pairwise RankNet loss over preference pairs \(\mathcal{P}\): $\(\mathcal{L}(q, \mathcal{D}_k) = \sum_{(i,j) \in \mathcal{P}} \log \left( 1 + \exp\left( -\frac{s_i - s_j}{\tau} \right) \right)\)$ where temperature \(\tau\) is set to \(1/8\), and preference pairs \((i,j) \in \mathcal{P}\) denote that candidate \(d_i\) was ranked above \(d_j\) by an expressive teacher reranker (such as Jina Reranker V3 or Qwen3-32B). Because relevance score \(s_i\) depends directly on compressed embedding \(c_i = f_{\theta_c}(d_i)\), ranking gradients flow unhindered from loss \(\mathcal{L}\) through decoder states \(h_i\) back into compressor parameters \(\theta_c\). This end-to-end gradient pathway forces the compressor to discard task-irrelevant lexical clutter and prioritize discriminative semantic features essential for precise relevance ranking.

Loss & Training

  • Backbone & Adaptation: The system employs Qwen-2.5 8B-instruct as the shared foundational backbone, equipping both the compressor and the decoder reranker with low-rank adaptation (LoRA) adapters;
  • Teacher Models & Data Formulation: Training signals are distilled without manual ground-truth labels. On MS MARCO (0.5M queries), the top-50 documents retrieved by SPLADE-V3 are scored by Jina Reranker V3 (a 0.6B model matching Qwen3-4B performance), with 16 documents randomly sampled per query. To scale capacity, an additional collection of 150K queries scored by zero-shot Qwen3-32B (derived from BGE-M3) is incorporated;
  • Optimization Parameters: Models are optimized for 2 epochs using AdamW with learning rate \(1 \times 10^{-4}\), a per-device batch size of 2 (16 documents per query), and 16 gradient accumulation steps. Total fine-tuning completes in 48 wall-clock hours on a single NVIDIA A100 GPU.

Key Experimental Results

Main Results

Evaluation is conducted across 12 datasets from the BEIR benchmark (excluding ArguAna) and TREC Deep Learning 2019/2020. The candidate pool consists of the top-50 documents retrieved by SPLADE-V3. Efficiency is benchmarked on a single NVIDIA A100 GPU and reported as the latency ratio relative to RRK (0.06 seconds per query).

Model Category Model Name Backbone Input Len BEIR Avg nDCG@10 Latency Ratio (โ†“) Seconds/Query (s/q)
Compressed Listwise (Ours) RRKโ€ -BGE 7B 512 58.9 1.0ร— 0.06
RRKโ€  (MS+E2R) 7B 512 58.4 1.0ร— 0.06
RRKโ€ -MS 7B 512 57.5 1.0ร— 0.06
Compressed Pointwise RRK-MS (Pointwise) 7B 512 57.5 3.5ร— 0.21
Public Rerankers Jina-v3โ€  (Teacher) 0.6B 512 59.8 4.7ร— 0.28
Qwen3-0.6B 0.6B 512 57.4 5.3ร— 0.32
Qwen3-4B 4B 512 60.3 20.8ร— 1.25
Uncompressed Baselines ModernBERT-Large 0.4B 512 57.2 2.2ร— 0.13
Qwen2.5-7B (Full text) 7B 512 59.7 21.0ร— 1.26

Note: โ€  indicates listwise reranking architectures; all benchmarks executed in identical PyTorch FlashAttention-2 fp16 environments.

Ablation Study

1. Compressor Training Strategy Ablation (BEIR 12 Datasets nDCG@10) Evaluating how the PISCO soft-compressor is initialized and adapted proves that task-specific fine-tuning is vital:

Compressor Configuration nDCG@10 Performance Impact
Frozen PISCO Compressor 55.5 Direct transfer of generic RAG compression incurs a 2.9 drop
Compressor Trained from Scratch 57.7 Joint training without PISCO initialization remains highly competitive
Fine-Tuned PISCO Compressor 58.4 Best performance via adapting pretrained compression latents to ranking

2. Long-Document Length Scaling & Latency Gains (MS MARCO Document DL19/DL20) On long-document corpora where raw document lengths range between 1000 and 2000 tokens, input compression yields staggering efficiency dividends:

Benchmark Document Length RRKโ€  (Ours) nDCG / Lat Jina-v3โ€  (0.6B) nDCG / Lat Qwen3-0.6B nDCG / Lat Qwen3-4B nDCG / Lat
TREC-DL19 (Doc) 512 70.6 / 1.0ร— 69.7 / 6.0ร— 63.0 / 13.0ร— 62.6 / 11.0ร—
1024 72.1 / 1.0ร— 70.0 / 7.0ร— 66.9 / 21.0ร— 66.4 / 24.0ร—
2048 72.0 / 1.0ร— 69.7 / 8.0ร— 70.0 / 37.0ร— 68.3 / 58.0ร—
TREC-DL20 (Doc) 512 67.7 / 1.0ร— 66.3 / 6.0ร— 58.5 / 13.0ร— 60.0 / 12.0ร—
1024 67.0 / 1.0ร— 66.4 / 7.0ร— 62.8 / 20.0ร— 62.6 / 24.0ร—
2048 68.6 / 1.0ร— 67.4 / 8.0ร— 66.9 / 35.0ร— 64.5 / 59.0ร—

Note: In DL19 and DL20 benchmarks, RRK achieves an effective compression factor of 256ร— at 2048 tokens, operating nearly 60ร— faster than Qwen3-4B.

Key Findings

  • Pareto Dominance Over Sub-Billion Models: Despite utilizing a 7B parameter foundation, RRK runs over 5ร— faster than 0.6B models (Qwen3-0.6B, Jina-v3) and 2ร— faster than ModernBERT (0.4B), demonstrating that shrinking sequence length via soft tokens is far more effective for inference throughput than reducing model parameters.
  • Robust Length Generalization on Long Documents: Although trained on 128-token passages, RRK transfers seamlessly to 2048-token documents without quality degradation, achieving 72.0 nDCG@10 on DL19 and outperforming public 4B rerankers by 3.7 points while running 58ร— faster.
  • Multi-Teacher Listwise Synergy: While the pointwise variant RRK-PW struggled to converge when training on heterogeneous teacher score distributions, the listwise formulation RRKโ€  seamlessly leverages ordinal ranking data from multiple teachers, climbing from 57.5 to 58.9 nDCG@10 when augmenting MS MARCO with BGE data.

Highlights & Insights

  • Cross-Task Compression Transfer: Demonstrates for the first time that continuous document representations optimized for generative RAG retain rich fine-grained signals capable of transferring directly to discriminative listwise ranking.
  • Architectural Elegance via Query Sandwiches: Employs an intuitive token sandwich layout to achieve full-candidate bidirectional contextuality inside standard causal autoregressive architectures without modifying lower-level attention kernels.
  • End-to-End Gradient Propagation: Unlocks direct gradient flow from the RankNet ranking objective back into the document compressor parameters, overcoming the representational ceiling of frozen first-stage retrieval embeddings.

Limitations & Future Work

  • Performance Decay on Lengthy Queries: RRK's efficiency gains stem from compressing candidate documents; if deployed on benchmarks featuring extensive prompt queries (e.g., BRIGHT, where queries can reach hundreds of tokens), the duplicate query tokens \(2|q|\) dilute the compression benefits.
  • Corpus Storage Overhead: Storing \(l=8\) floating-point hidden states (\(d_{\text{model}}=3584\)) per document requires approximately 230 GB of storage for the 8.8M passages in MS MARCO, necessitating future exploration of vector quantization (PQ/SQ) to reduce disk footprint.
  • Underperformance on Sub-4B Backbones: Initial attempts by the authors to build RRK on smaller 1Bโ€“4B parameter models resulted in noticeable capacity degradation, indicating that high-fidelity dual compression-and-ranking capabilities currently demand larger parameter spaces.
  • vs PE-Rank (Liu et al., 2025a): PE-Rank relies on single-token embeddings inherited from first-stage dense retrievers and requires sequential token generation to output document IDs; RRK utilizes 8-token soft LLM compression and parallel cosine scoring, surpassing PE-Rank by over 5 nDCG@10 points while running faster.
  • vs E2Rank (Liu et al., 2026): E2Rank shares embeddings between retrieval and reranking but concatenates raw text from the top-20 documents to construct its query representation, remaining bottlenecked by text processing; RRK processes all candidate inputs in compressed form, achieving higher speed and comparable effectiveness.

Rating

  • Novelty: โญโญโญโญโ˜† [Pioneers the application of multi-token soft prompt compression to listwise LLM reranking]
  • Experimental Thoroughness: โญโญโญโญโญ [Exhaustive evaluation across 12 BEIR datasets, long-document TREC tracks, and detailed ablation studies]
  • Writing Quality: โญโญโญโญโญ [Clear mathematical formulations, clean architectural diagrams, and compelling empirical exposition]
  • Value: โญโญโญโญโญ [Provides a practical blueprint for deploying 7B-parameter listwise rerankers under strict industrial latency budgets]