Skip to content

QReason: Query-Focused Decoupled Chain-of-Thought for Efficient Passage Reranking

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Passage Reranking, Decoupled Chain-of-Thought, Reinforcement Learning, Query Rewriting, Listwise Reranking

TL;DR

QReason addresses the severe Chain-of-Thought redundancy and high latency bottleneck caused by sliding-window strategies in listwise LLM rerankers by decoupling global query-focused reasoning from window-specific passage relevance assessment, leveraging a two-stage training scheme (Relevance-Grounded Supervised Tuning and Rank-Aligned Dual-Reward Reinforcement Learning) to generate a one-time reusable reasoning query that matches or outperforms full reasoning rerankers at up to 10.75x speedup.

Background & Motivation

Passage reranking constitutes a pivotal stage in modern Information Retrieval (IR) pipelines, tasked with refining the initial ranking of candidate passages retrieved by first-stage dense or sparse models to maximize top-tier relevance precision. With the advent of large language models, listwise reranking frameworks leveraging all-to-all attention have established state-of-the-art results through joint contextual modeling of relative passage interactions. To tackle reasoning-intensive retrieval tasks where relevant documents share non-trivial, multi-hop semantic relationships with queries, recent approaches such as ReasonRank, Rank1, and Rank-R1 introduce explicit Chain-of-Thought (CoT) reasoning before emitting the permutation order, achieving marked gains in semantic disambiguation.

However, constrained by finite context windows and permutation combinatorial complexity, listwise LLM rerankers inevitably rely on overlapping sliding-window strategies to process long candidate lists. Through quantitative empirical probing of ReasonRank on the BRIGHT benchmark, the authors uncover a fundamental architectural inefficiency: across successive overlapping sliding windows for the same query, the independently generated reasoning traces exhibit an average semantic similarity exceeding 90% (measured via BGE-m3 text embeddings). This indicates that standard reasoning rerankers repeatedly recompute global query intent interpretation and hypothetical evidence facets across every single window, which are largely invariant to the local candidate passages inside each window, thereby inducing massive computational redundancy and unacceptable inference latency.

Meanwhile, directly applying conventional retrieval-stage query rewriting models (such as GPT-4 expansions or TongSearch) fails in precision-oriented reranking pipelines, as unconstrained expansions often introduce extraneous tokens and semantic noise that misguide fine-grained query-document interactions. To overcome these dual dilemmas, this work proposes to externalize and decouple global query reasoning from window-specific ranking. Core idea: train a specialized rewriter via relevance-grounded supervised tuning on gold passages followed by dual-reward reinforcement learning with listwise NDCG and pointwise relevance constraints, producing a single, grounded reasoning query that is reused across all sliding windows by an efficient non-reasoning reranker.

Method

Overall Architecture

Conventional listwise reasoning rerankers repeatedly infer both reasoning trace \(c_j = g(q, W_j)\) and local permutation \(\pi_j = f(q, W_j, c_j)\) for every sliding window \(W_j\). In contrast, QReason cleanly decouples this pipeline into query rewriting and non-reasoning window ranking: a dedicated rewriter \(RW(\cdot)\) processes the original query \(q\) once to generate a ranking-oriented reasoning query \(q' = RW(q)\), which is directly concatenated to \(q\) and reused across all \(K\) sliding windows by a fixed non-reasoning reranker \(f\), yielding \(\pi_j = f(q, W_j, q')\). To endow the rewriter with strong relevance grounding and task-specific ranking capability without test-time passage leakage, QReason employs a two-stage optimization pipeline comprising Relevance-Grounded Supervised Tuning (RGST) and Rank-Aligned Dual-Reward Refinement (RADR).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Query and Candidate Pool"] --> B["Relevance-Grounded Supervised Tuning<br/>Distilling teacher CoT with gold passages"]
    B --> C["Rank-Aligned Dual-Reward Refinement<br/>Joint listwise NDCG and pointwise relevance GRPO"]
    C --> D["Single-Pass Reasoning Query Rewriter<br/>Emit global intent guide q' once"]
    D --> E["Cross-Window Non-Reasoning Reranking<br/>Reuse q' across sliding windows efficiently"]

Key Designs

1. Relevance-Grounded Supervised Tuning: Eliminating semantic hallucination and unconstrained expansion drift in query reasoning
Standard query rewriting approaches optimized for broad retrieval recall tend to inject extraneous entity associations and speculative details that compromise high-precision reranking. In the RGST stage, QReason leverages gold relevant passages \(P^+\) as supervisory anchors to define the critical semantic facets necessary for accurate discrimination. A teacher reasoning model (DeepSeek-R1) takes \((q, P^+)\) as input to synthesize a query-focused reasoning query \(q'\) that traces the explicit logical connection between question intent and target evidence. The rewriter (initialized from Qwen2.5-7B-Instruct) is then fine-tuned on input-output pairs \(([q; P^+], q')\) via causal language modeling: $$ \mathcal{L}{\text{RGST}}(\theta) = - \sum \log p_\theta(q'}^{|q'|t \mid q, P^+, q') $$ This supervised phase provides a stable, grounded initialization where the model internalizes the structural reasoning patterns characteristic of true positive documents before transitioning to independent generation.

2. Rank-Aligned Dual-Reward Refinement: Bridging training-inference discrepancies and preventing single-reranker overfitting
At inference time, gold relevant passages \(P^+\) are strictly absent, creating a distribution mismatch with RGST; moreover, pure supervised likelihood does not optimize downstream ranking permutation metrics. In the RADR stage, QReason hides \(P^+\) from the policy input and applies Group Relative Policy Optimization (GRPO) over sampled rewrite candidates \(G(q) = \{q'_1, \dots, q'_{|G|}\} \sim \pi_\theta(\cdot \mid q)\). To avoid degenerate rewrites that game the scoring idiosyncrasies of a fixed downstream reranker, RADR couples two complementary rewards: a listwise reward \(\text{List}(q'_j) = \text{NDCG@10}(\pi_j, Y_q)\) evaluating the final permutation produced by a downstream non-reasoning reranker, and a pointwise reward \(\text{Point}(q'_j) = \frac{1}{N_q} \sum_{i=1}^{N_q} \text{is\_rel}(q'_j, p^+_{q,i})\) where an independent LLM judge verifies semantic consistency between \(q'_j\) and each gold passage \(p^+_{q,i}\). The composite reward is formulated as: $$ r_j = \text{List}(q'_j) + \lambda \text{Point}(q'_j) $$ Balancing listwise permutation quality with pointwise evidence faithfulness ensures the rewriter produces robust, transferable, and hallucination-free reasoning expansions.

3. Cross-Window Non-Reasoning Reranking: Eliminating autoregressive decoding bottlenecks across sliding windows
By externalizing the query reasoning phase into a one-time rewrite step, the downstream scoring across sliding windows transforms entirely into non-reasoning execution. For a candidate pool segmented into \(K\) sliding windows, conventional reasoning rerankers incur a total generation token cost of \(G_{\text{reason}} \approx K(R + O)\), where \(R\) and \(O\) denote reasoning and ordering output token lengths. QReason compresses this generation footprint to \(G_{\text{QReason}} \approx R' + K O\). Although appending \(q'\) slightly lengthens input prompts during the prefilling stage, it completely bypasses the repetitive, latency-prohibitive autoregressive generation of CoT traces across \(K-1\) windows, shifting computational load from sequential decoding to highly parallel matrix operations.

Loss & Training

During the RADR reinforcement learning stage, the policy is optimized using the clipped surrogate GRPO objective with per-token Kullback-Leibler (KL) regularization against reference policy \(\pi_{\text{ref}}\): $$ \mathcal{L}{\text{RADR}}(\theta) = - \frac{1}{|G|} \sum{|q'}^{|G|} \frac{1j|} \sum^{|q'j|} \left[ \min\left( \rho}(\theta) \hat{Aj, \, \text{clip}(\rho}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{Aj \right) \right] - \beta D)) $$ where }}(\pi_\theta(\cdot \mid s_{j,t}) \parallel \pi_{\text{ref}}(\cdot \mid s_{j,t\(\rho_{j,t}(\theta) = \frac{\pi_\theta(q'_{j,t} \mid s_{j,t})}{\pi_{\theta_{\text{old}}}(q'_{j,t} \mid s_{j,t})}\) is the importance sampling probability ratio, \(\hat{A}_j\) is the group-normalized relative advantage, and \(\beta = 0.001\). RGST fine-tuning is conducted on 8 A800 GPUs for 3 epochs (learning rate \(5 \times 10^{-6}\), taking \(\sim\)2 hours). RADR runs for \(\sim\)7 hours across 8 A800 GPUs (\(|G|=8\), sampling temperature 0.8, learning rate \(1 \times 10^{-6}\)) with reward evaluation servers hosting Qwen3-30B-A3B (reranker) and Qwen3-8B (pointwise judge) on 2 H100 GPUs via vLLM.

Key Experimental Results

Main Results

On the challenging BRIGHT benchmark (12 diverse reasoning-intensive domains including economics, biology, LeetCode, AoPS, and theorem proving), candidate top-100 passages retrieved by ReasonIR (8B) are reranked using a sliding window size of 20 and step size of 10. The primary NDCG@10 comparison is summarized below:

Model / Reranker Setting Architecture Econ. Bio. Leet. AoPS TheoQ. Avg NDCG@10
ReasonIR (Initial Retriever) 8B 32.43 43.49 32.33 8.14 32.84 30.66
RankT5 (Pointwise Non-Reason) 3B 11.28 11.59 27.41 9.33 18.27 16.75
RankZephyr (Listwise Non-Reason) 7B 19.85 33.77 29.14 6.61 24.90 22.49
Rearank (Listwise Reasoning) 7B 30.46 47.08 31.25 7.76 30.17 31.89
Rank1 (Pointwise Reasoning) 32B 25.63 42.72 12.24 5.67 29.64 28.00
Rank-K (Listwise Reasoning) 32B 28.49 48.22 25.98 9.25 35.93 31.95
ReasonRank (Listwise Reasoning) 32B 34.49 54.65 27.77 7.94 38.49 36.45
Qwen2.5-32B + Original Query 32B 32.27 52.18 17.82 7.16 27.59 32.75
Qwen2.5-32B + GPT-4 Rewrite 32B 30.59 52.72 22.34 6.80 35.35 32.98 (+0.23)
Qwen2.5-32B + TongSearch (7B) 32B 30.08 49.27 20.11 7.48 28.67 32.14 (-0.61)
Qwen2.5-32B + QReason 7B+32B 35.70 51.46 18.06 9.66 32.51 35.18 (+2.43)
Qwen3.5-35B-A3B + Original Query 35B-A3B 32.07 53.89 28.75 8.26 40.28 35.49
Qwen3.5-35B-A3B + TongSearch (7B) 35B-A3B 33.84 53.27 30.72 8.47 40.95 35.69 (+0.20)
Qwen3.5-35B-A3B + QReason 7B+35B 35.34 56.44 31.55 9.17 41.73 36.93 (+1.44)

In end-to-end latency evaluations across all 12 BRIGHT datasets, QReason paired with Qwen3.5-35B-A3B requires an average of only 212.44 seconds per run compared to 1542.49 seconds for ReasonRank (32B), delivering a 7.26x overall speedup (reaching 10.75x speedup on LeetCode). The one-time query rewriting stage takes only 23.76 seconds on average, representing a negligible 3.9% of total pipeline latency.

Ablation Study

Systematic component ablations on the full BRIGHT benchmark utilizing Qwen2.5-32B-Instruct as the downstream non-reasoning reranker:

Config Avg NDCG@10 Delta vs. Full Note
QReason Full Model 35.18 - Complete two-stage pipeline (RGST + RADR)
Only RGST 33.50 -1.68 SFT on gold evidence only; lacks RL ranking alignment
Only RADR 33.91 -1.27 RL fine-tuning without evidence-grounded cold-start initialization
RGST w/o \(P^+\) Guidance 32.98 -2.20 Standard SFT without gold passages; severe hallucination & drift
RADR w/o Pointwise Reward 33.85 -1.33 Removing relevance constraints leads to reranker score hacking
RADR w/o Listwise Reward 33.90 -1.28 Removing NDCG reward decouples generation from ranking objective

Key Findings

  • Relevance Guidance (\(P^+\)) is the Core Anchor of Grounding: Omitting gold passage context during the initial SFT stage causes the steepest performance collapse (-2.20 points), demonstrating that ungrounded reasoning generation easily drifts toward misleading keywords that confuse downstream rerankers.
  • Dual Rewards Prevent Policy Exploitation: Optimizing purely for the listwise NDCG reward causes the policy to exploit scoring quirks of the fixed reranker with synthetic phrasing, whereas optimizing purely for pointwise relevance fails to capture global listwise ranking dynamics. Performance peaks precisely at \(\lambda = 0.6\).
  • Robust Transferability across Model Scales and Benchmarks: QReason consistently boosts diverse general LLM rerankers, achieving +1.61 to +3.24 NDCG@10 gains on 70B+ models (Qwen2.5-72B, Llama3.3-70B, Qwen3.6-35B-A3B) and +1.47 to +2.31 on smaller 7B-14B models. It also delivers consistent improvements on FreshStack (technical code retrieval, reaching 33.63 NDCG@10) and NeuCLIRBench (multilingual and cross-lingual retrieval, reaching 31.61 NDCG@10).

Highlights & Insights

  • Decoupling Redundant Computational Topologies: By pinpointing that over 90% of CoT content across sliding windows represents redundant global intent analysis, the paper replaces \(K-1\) sequential autoregressive generation cycles with a single 7B rewrite, providing an elegant and practical blueprint for high-throughput IR systems.
  • Gold Passages as Discardable Scaffolding: Utilizing gold passages during training as semantic supervision anchors and dropping them completely at test time enables the lightweight rewriter to emulate deep evidence-seeking intuition entirely autonomously.
  • Rethinking Retrieval vs. Reranking Query Modification: Demonstrates why popular recall-oriented query expanders fail under precision-sensitive listwise reranking, establishing the principle that reranking rewrites must optimize listwise discrimination rather than unconstrained coverage.

Limitations & Future Work

  • Lack of Multi-Dimensional Granular Quality Rubrics: The dual-reward formulation models overall NDCG and binary relevance, but lacks explicit constraints on factual faithfulness, facet coverage, and information density. Future extensions could incorporate structured rubric rewards.
  • Reasoning Depth on Extreme Multi-Step Mathematical Proofs: While QReason shows strong results on LeetCode and AoPS, a 7B rewriter generating a single concise query may still miss intricate symbolic steps on highly complex formal proofs.
  • Prefill Context Length Growth: Appending the reasoning trace slightly increases the prefill token length for each window. Although prefill is highly parallelized, exploring compressed latent representations or prefix embeddings could further optimize memory footprints.
  • vs. ReasonRank / Rank-K: These methods mandate that the reranker generate multi-hundred-token reasoning traces for each sliding window sequentially, resulting in severe latency bottlenecks. QReason decouples the reasoning step, achieving higher accuracy than ReasonRank (32B) (36.93 vs. 36.45) while slashing latency by over 7x.
  • vs. DeepRetrieval / TongSearch: Prior RL-based rewriters are trained to optimize recall in bi-encoder first-stage retrieval, causing them to generate noisy keywords that degrade downstream listwise rerankers (e.g., TongSearch drops Qwen2.5-32B performance by 0.61 points). QReason directly aligns query generation with the reranking objective, ensuring clean and robust transferability.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Decoupling global query reasoning from sliding-window reranking is conceptually elegant and addresses a major pain point in LLM-based IR.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across BRIGHT, FreshStack, and NeuCLIRBench with 10+ backbone models, rigorous latency profiling, and meticulous ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined methodology exposition, and transparent mathematical formulations.
  • Value: ⭐⭐⭐⭐⭐ Immense practical utility for industrial search engines and academic research seeking high-accuracy, low-latency reasoning-enhanced reranking.