DuoPRAG: Rank-Disentangled and Gated Knowledge-Reasoning Fusion¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/naknak-choi/DuoPRAG
Area: Information Retrieval
Keywords: Parametric RAG, LoRA Fusion, Latent Reasoning, Rank-level Disentanglement, Multi-hop QA
TL;DR¶
DuoPRAG disentangles retrieved factual knowledge from multi-step reasoning into two distinct LoRA adapters combined via static rank masks and dynamic token-level gates, replacing brittle lexical CoT prompts with continuous latent reasoning tokens to achieve state-of-the-art multi-hop QA performance in a query-only regime.
Background & Motivation¶
Large language models (LLMs) have achieved remarkable success on knowledge-intensive question answering benchmarks, but updating their parameters to reflect real-world facts incurs prohibitive retraining and fine-tuning expenses. While standard retrieval-augmented generation (RAG) augments input prompts by concatenating retrieved passages, it suffers from bounded context lengths, substantial generation latencies, and frequent knowledge conflicts between parametric memory and external text. To overcome these limitations, Parametric RAG (PRAG and DyPRAG) has emerged as an appealing paradigm that translates retrieved documents into low-rank adapter (LoRA) weights and injects them directly into the feed-forward layers of the LLM, enabling passage-free, query-only generation at inference.
However, existing parametric RAG architectures suffer from a critical conceptual entanglement: they optimize per-document LoRA adapters on training prompts containing both retrieved passages and lexical Chain-of-Thought (CoT) exemplars, conditioning the loss on the entire generated rationale-answer sequence. As a result, the very same low-rank parameter space is forced to simultaneously memorize document-specific facts and procedural step-by-step reasoning. At inference time, relying on explicit surface-level CoT triggers creates extreme demonstration sensitivity, inducing three acute failure modes: severe over-generation, entity leakage from few-shot exemplars, and answer format mismatch. As empirically demonstrated in the paper, removing answer-extraction heuristics causes PRAG's average F1 across four multi-hop QA datasets to collapse catastrophically from 27.97% to 14.23%.
Furthermore, simply splitting the network into independent LoRA adapters does not ensure functional separation. Centered Kernel Alignment (CKA) similarity analysis shows that naive linear addition of independently trained knowledge and reasoning LoRAs yields an alarming 0.91 representation overlap at the output layer despite near-zero cosine parameter similarity, indicating severe functional redundancy. Parametric knowledge injection without learned routing fails to resolve CoT demonstration sensitivity and degrades output fidelity. Core idea: completely disentangle parametric knowledge injection from procedural multi-step reasoning into two specialized LoRA branches, shift reasoning from brittle surface CoT text to continuous latent tokens, and fuse them via layer-wise static rank masks and token-conditional dynamic gates to deliver robust, query-only multi-hop inference.
Method¶
Overall Architecture¶
DuoPRAG decouples parametric context injection and multi-step inference into two functionally orthogonal branches that interact through a rank-disentangled routing mechanism. Given an input question \(q\) and a set of retrieved documents \(P = \{p_1, \dots, p_N\}\), the knowledge branch (\(W_{knw}\)) maps the documents into factorized parameter updates that inject retrieved evidence into the base LLM's MLP layers. Simultaneously, the reasoning branch (\(W_{rsn}\)) guides the model to autoregressively synthesize \(K\) continuous latent reasoning vectors (\(z_1, \dots, z_K\)) between learnable boundary tokens \(\langle bot \rangle\) and \(\langle eot \rangle\). At each MLP module, the rank-space activations of both branches are dynamically modulated by a static layer-level rank mask and an input-dependent token-level gate before adding to the frozen base model's representations, allowing the model to decode the exact final answer without verbalizing intermediate reasoning tokens or receiving retrieved passages in its prompt.
The end-to-end processing pipeline and module interactions are illustrated in the flowchart below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Question q & Retrieved Documents P"] --> B1["Parametric Knowledge Disentanglement<br/>Hypernetwork encodes passages into factorized LoRA"]
In --> B2["Implicit Latent-Space Reasoning<br/>Autoregressive generation of K continuous latent tokens"]
B1 --> F["Rank Mask and Token Gate Fusion<br/>Modulation via static rank mask & dynamic gate"]
B2 --> F
F --> Out["Output: Direct decoding of final answer a"]
Key Designs¶
1. Parametric Knowledge Disentanglement: Isolating Evidence Ingestion from Reasoning Behavior Prior parametric RAG approaches train document LoRA modules using prompt templates that bundle few-shot CoT exemplars, causing factual knowledge to be corrupted by prompt format dependencies. DuoPRAG constructs a dedicated Knowledge LoRA branch \(W_{knw}\) that strictly encodes retrieved passages \(P\) into low-rank weight updates across the base LLM's MLP modules. The framework remains agnostic to the underlying parameter generator \(\Phi\), seamlessly accommodating offline single-document adapters as well as dynamic hypernetworks such as DyPRAG or Doc-to-LoRA. Specifically, for each retrieved passage \(p_i\), the generator outputs low-rank factors \((A_{knw, i}, B_{knw, i}) \in \mathbb{R}^{r_{knw} \times d_{in}} \times \mathbb{R}^{d_{out} \times r_{knw}}\). To preserve the factorized parameter form required for subsequent rank-level composition, DuoPRAG aggregates multiple passage adapters via factor-wise averaging at each module: $\(A_{knw} = \frac{1}{N} \sum_{i=1}^N A_{knw, i}, \quad B_{knw} = \frac{1}{N} \sum_{i=1}^N B_{knw, i}\)$ This factor-level aggregation preserves the compact rank budget (\(r_{knw}=2\)) and ensures that the knowledge adapter represents pure factual evidence devoid of any lexical demonstration bias.
2. Implicit Latent-Space Reasoning: Replacing Verbose CoT Text with Continuous Latent Tokens Relying on lexical CoT to verbalize intermediate deductions increases sequence lengths and causes prompt demonstration leakage. DuoPRAG eliminates surface-level CoT by transferring multi-step reasoning into continuous latent representations. The model defines learnable boundary tokens \(\langle bot \rangle\) and \(\langle eot \rangle\), inserting \(K\) continuous latent vectors \(z_1, \dots, z_K \in \mathbb{R}^d\) (\(K=4\) by default) between the input question and the answer sequence. At step \(k\), given the prefix \(pre_{k-1} = [q; \langle bot \rangle; z_1; \dots; z_{k-1}]\), the \(k\)-th continuous latent token is generated by passing the final-layer hidden state \(h_{k-1}\) through a two-layer MLP projector \(prj\): $\(z_k = prj(h_{k-1}) = W_2 \text{GELU}(W_1 h_{k-1})\)$ After producing all \(K\) latent tokens, the closing delimiter \(\langle eot \rangle\) is appended and the model directly decodes the final answer \(a\). To ground each latent vector in meaningful semantic deduction rather than uninterpretable noise, an auxiliary decoder \(D_\eta\) is introduced during training. \(D_\eta\) treats each \(z_k\) as a soft prompt prefix and autoregressively reconstructs the corresponding natural language reasoning step. Its gradients update only \(z_k\) and a lightweight adapter without backpropagating into the frozen base model. Crucially, \(D_\eta\) is completely discarded at inference time, enabling direct, unpolluted answer generation in just 4 latent computation steps.
3. Rank Mask and Token Gate Fusion: Resolving Functional Redundancy in Low-Rank Subspaces Summing independently trained knowledge and reasoning LoRA adapters results in severe representational collisions, exhibiting up to 0.91 CKA feature overlap. To enforce true functional disentanglement, DuoPRAG introduces a dual-routing mechanism that operates across individual rank dimensions. For each Transformer layer \(l\) and branch \(b \in \{knw, rsn\}\), a learnable static rank mask vector \(M^{(l)}_b \in \mathbb{R}^{r_b}\) is initialized to all ones, establishing a baseline structural allocation of rank capacity. Concurrently, for each MLP projection module \((l, t)\) where \(t \in \{gate, up, down\}\), a linear gate head \(W_{g, b}^{(l, t)} \in \mathbb{R}^{r_b \times d_{in}}\) calculates an input-dependent, token-level activation score based on the incoming hidden state \(x_\tau\): $\(g^{(l, t)}_b(x_\tau) = \sigma(W_{g, b}^{(l, t)} x_\tau) \in \mathbb{R}^{r_b}\)$ The static mask and dynamic gate are combined via element-wise multiplication into a unified routing vector \(R^{(l, t)}_{b, \tau} = M^{(l)}_b \odot g^{(l, t)}_b(x_\tau)\), which scales the rank activation \(A^{(l, t)}_b x_\tau\) prior to the up-projection: $\(\Delta^{(l, t)}_b(x_\tau) = \alpha_b B^{(l, t)}_b (R^{(l, t)}_{b, \tau} \odot A^{(l, t)}_b x_\tau), \quad out(x_\tau) = base(x_\tau) + \Delta^{(l, t)}_{knw}(x_\tau) + \Delta^{(l, t)}_{rsn}(x_\tau)\)$ This multiplicative formulation enables each rank dimension to dynamically prioritize factual recall or logical deduction on a per-token basis, crushing the inter-branch CKA similarity from 0.910 down to 0.031.
Loss & Training¶
DuoPRAG is optimized across two distinct stages:
- Stage 1: Reasoning-Only Pretraining. The reasoning branch \(W_{rsn}\), projector \(prj\), boundary embeddings, and auxiliary decoder \(D_\eta\) are trained from scratch on 44,665 single-passage QA instances (SQuAD, DROP, PWC, ROPES, NQ) without retrieval, learning to perform implicit reasoning over latent tokens.
- Stage 2: Joint LoRA Training. The base LLM and knowledge generator \(\Phi\) remain frozen. Initializing from Stage 1, the knowledge adapter \(W_{knw} = \Phi(P)\), rank masks \(M\), and gate heads \(W_g\) are introduced. The dual-LoRA architecture is jointly trained on 1,799 multi-hop QA instances (2WikiMHQA, HotpotQA, PopQA, CWQ).
Both stages optimize a composite objective: $\(\mathcal{L} = \lambda_{ans} \mathcal{L}_{ans} + \lambda_{step} \mathcal{L}_{step}\)$ with \(\lambda_{ans} = \lambda_{step} = 1\). The answer loss \(\mathcal{L}_{ans}\) is the cross-entropy over target answer tokens conditioned on the latent prefix, while the step supervision loss \(\mathcal{L}_{step}\) trains \(D_\eta\) to decode reference reasoning text \(s_k = (y_{k, 1}, \dots, y_{k, L_k})\) from each latent vector \(z_k\): $\(\mathcal{L}_{step} = - \sum_{k=1}^{\min(K, N_{step})} \sum_{t=1}^{L_k} \log p_\eta(y_{k, t} \mid z_k, y_{k, <t})\)$ The reference reasoning chains are pre-generated offline by Qwen3.5-27B to provide reliable step-level supervision.
Key Experimental Results¶
Main Results¶
Evaluation was conducted across three instruction-tuned backbones (LLaMA-3.2-1B, Qwen2.5-1.5B, and LLaMA-3-8B) on both multi-hop reasoning benchmarks (2WikiMultiHopQA, HotpotQA) and single-hop entity benchmarks (PopQA, ComplexWebQuestions). Performance is measured by Exact Match (EM %) and token-level F1 (%):
| Backbone Model | Method | 2WikiMHQA | HotpotQA | PopQA | CWQ | Average (Avg) |
|---|---|---|---|---|---|---|
| LLaMA-3.2-1B | Vanilla | 17.47 / 22.87 | 18.56 / 24.10 | 0.67 / 2.26 | 23.67 / 34.94 | 16.74 / 21.04 |
| PRAG-woC | 20.13 / 25.92 | 19.00 / 25.35 | 12.00 / 23.58 | 26.00 / 35.86 | 19.57 / 26.51 | |
| DyPRAG-woC | 24.27 / 29.91 | 19.56 / 25.97 | 7.33 / 11.33 | 28.33 / 36.86 | 21.57 / 27.57 | |
| DuoPRAG-woC (Ours) | 29.93 / 37.17 | 21.22 / 29.26 | 30.67 / 30.67 | 36.00 / 46.08 | 29.46 / 35.79 | |
| Standard RAG (Context) | 17.93 / 24.77 | 21.44 / 30.33 | 9.67 / 17.65 | 25.67 / 37.39 | 18.93 / 26.99 | |
| DuoPRAG (Context) | 32.07 / 40.06 | 27.45 / 35.71 | 53.33 / 54.17 | 39.00 / 47.58 | 37.96 / 44.38 | |
| Qwen2.5-1.5B | Vanilla | 20.87 / 27.20 | 14.78 / 23.13 | 0.67 / 2.87 | 18.00 / 26.47 | 16.74 / 25.79 |
| PRAG-woC | 21.93 / 29.38 | 16.00 / 24.04 | 1.33 / 3.87 | 22.31 / 30.82 | 18.13 / 25.37 | |
| DyPRAG-woC | 21.87 / 28.46 | 17.11 / 24.93 | 3.00 / 6.64 | 22.67 / 31.94 | 18.64 / 25.56 | |
| DuoPRAG-woC (Ours) | 29.47 / 36.39 | 21.22 / 28.76 | 30.33 / 30.33 | 36.67 / 45.95 | 29.42 / 35.36 | |
| Standard RAG (Context) | 16.33 / 23.89 | 14.89 / 24.68 | 0.67 / 9.97 | 18.64 / 28.23 | 14.56 / 23.17 | |
| DuoPRAG (Context) | 28.73 / 36.28 | 31.22 / 40.93 | 51.33 / 52.00 | 41.33 / 49.44 | 38.15 / 44.66 | |
| LLaMA-3-8B | Vanilla | 30.00 / 36.43 | 19.89 / 28.64 | 4.67 / 7.96 | 30.00 / 42.44 | 24.43 / 31.85 |
| PRAG-woC | 33.20 / 40.54 | 35.55 / 45.88 | 20.33 / 26.13 | 32.67 / 43.54 | 32.57 / 41.00 | |
| DyPRAG-woC | 32.07 / 39.17 | 24.67 / 37.33 | 11.00 / 13.60 | 32.67 / 41.87 | 27.80 / 36.23 | |
| DuoPRAG-woC (Ours) | 38.40 / 45.00 | 31.78 / 39.71 | 42.33 / 42.56 | 39.67 / 49.62 | 38.04 / 44.22 | |
| Standard RAG (Context) | 28.40 / 34.20 | 19.13 / 28.67 | 5.67 / 16.13 | 25.33 / 35.45 | 23.04 / 30.86 | |
| DuoPRAG (Context) | 38.93 / 46.82 | 38.44 / 49.52 | 52.00 / 53.02 | 43.67 / 54.18 | 43.26 / 50.89 |
Ablation Study¶
The table below details the performance of different fusion configurations on LLaMA-3.2-1B, evaluating how different masking and gating structures impact accuracy and representation disentanglement (CKA):
| Configuration | 2WikiMHQA | HotpotQA | PopQA | CWQ | Average (EM / F1 %) | Feature CKA (โ) |
|---|---|---|---|---|---|---|
| Full DuoPRAG | 29.93 / 37.17 | 21.22 / 29.26 | 30.67 / 30.67 | 36.00 / 46.08 | 29.46 / 35.79 | 0.031 |
| Mask only | 24.40 / 31.63 | 21.33 / 28.61 | 15.33 / 16.39 | 32.67 / 40.97 | 23.43 / 29.40 | 0.917 |
| Gate only | 27.47 / 35.19 | 20.22 / 27.43 | 25.00 / 25.89 | 35.33 / 42.56 | 27.00 / 32.77 | 0.105 |
| Scalar gate | 28.07 / 35.82 | 20.89 / 27.58 | 24.00 / 24.89 | 35.00 / 42.05 | 26.99 / 32.58 | โ |
| Token-independent gate | 26.40 / 34.13 | 20.55 / 27.77 | 23.33 / 24.22 | 35.33 / 42.37 | 26.40 / 32.12 | โ |
| Parameter-matched Additive | 34.00 / 39.63 | 21.11 / 26.99 | 20.33 / 20.33 | 32.67 / 42.42 | 27.03 / 32.34 | โ |
| Naive Linear Addition | 20.73 / 28.44 | 15.22 / 21.34 | 2.00 / 6.63 | 26.67 / 35.95 | 16.16 / 23.09 | 0.910 |
Key Findings¶
- Static and Dynamic Fusion are Highly Complementary: Omitting the token-conditional gate (Mask only) drops the average F1 by 6.39 points while exhibiting a high CKA similarity of 0.917. Conversely, removing the rank mask (Gate only) decreases F1 by 3.02 points. Only their combination achieves deep feature disentanglement (CKA = 0.031) and peak accuracy.
- Superiority Over Parameter-Matched Additive Fusion: The parameter-controlled additive variant trails DuoPRAG by 3.45 F1 points overall and exhibits severe instability on PopQA (20.33 vs. 30.67), proving that gains stem from the multiplicative rank-routing architecture rather than expanded parameter capacity.
- Massive Latency Reduction: Under an identical 128-token cap on 2WikiMHQA, PRAG and DyPRAG overgenerate 72 to 128 tokens, taking 1,533 ms and 2,506 ms per query. In contrast, DuoPRAG spends 255 ms on latent computation and 120 ms to emit a 4-token answer, achieving a total latency of 375 ms (a 4.1ร to 6.7ร speedup).
Highlights & Insights¶
- Parameter Orthogonality vs. Functional Redundancy: The paper uncovers a vital insight for LoRA composition: two adapter matrices can have zero cosine parameter similarity yet exhibit 91% CKA representational redundancy. Explicit routing along rank dimensions is necessary to enforce true functional specialization.
- Complete Immunity to CoT Demonstrations: By encoding reasoning steps into continuous latent vectors rather than text tokens, DuoPRAG becomes entirely immune to noisy, corrupt, or adversarial surface CoT demonstrations, rendering multi-hop inference impervious to formatting shifts.
- Plug-and-Play Hypernetwork Compatibility: The dual-LoRA framework generalizes seamlessly across distinct knowledge generators, yielding consistent gains whether paired with offline PRAG, DyPRAG, or Doc-to-LoRA.
Limitations & Future Work¶
- Bottlenecked by Document Compression Fidelity: On specialized factual domains such as MedMCQA and OpenBookQA, DuoPRAG-woC lags behind in-context Standard RAG, indicating that current hypernetworks lose fine-grained entity details when compressing documents into low-rank weights.
- Restricted Evaluation Domain: Experiments are limited to open-domain QA. Validating the dual-LoRA fusion mechanism on code generation, multi-step math (e.g. GSM8K, MATH), or agent tool-calling trajectories remains open for future research.
- Potential Extensions: Designing higher-capacity, entity-preserving parameter generators and integrating adaptive latent token budgets (\(K\)) conditioned on question difficulty represent promising research avenues.
Related Work & Insights¶
- vs. PRAG / DyPRAG: Prior parametric RAG methods entangle knowledge and lexical CoT into a single LoRA, suffering from output over-generation and demonstration leakage. DuoPRAG cleanly decouples knowledge and reasoning into distinct adapters, eliminating reliance on prompt-based CoT.
- vs. SIM-CoT: While SIM-CoT demonstrates the feasibility of continuous latent reasoning, DuoPRAG extends this principle to parametric RAG, addressing the critical challenge of composing dynamic document adapters with latent reasoning modules.
- vs. DuoLoRA: DuoLoRA introduced ZipRank for text-to-image style-content separation. DuoPRAG adapts rank-level disentanglement to autoregressive LLMs by introducing token-conditional gating to handle sequential language context.
Rating¶
- Novelty: โญโญโญโญโญ [First to decouple knowledge and reasoning into separate LoRA adapters with rank-level gated fusion in parametric RAG]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 3 base LMs, in-domain/OOD benchmarks, CKA analyses, ablation variants, and latency breakdowns]
- Writing Quality: โญโญโญโญโญ [Clear motivation, rigorous mathematical formulation, and compelling empirical diagnoses of existing failure modes]
- Value: โญโญโญโญโญ [Provides an elegant and practical blueprint for accelerating RAG inference and resolving prompt sensitivity in production LLM systems]