Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Reinforcement learning, Parametric knowledge retrieval, Hierarchical traversal, On-policy exploration, Layerwise representation analysis
TL;DR¶
This work demonstrates that reinforcement learning (RL) improves factual recall in LLMs not by injecting new facts, but by internalizing a systematic hierarchical traversal policy over existing latent parametric knowledge; through relation-disjoint knowledge-injection experiments and layerwise representational probing, the authors prove that iterated on-policy exploration reshapes question-navigation processing while leaving factual memory representations intact.
Background & Motivation¶
In large language model (LLM) post-training, reinforcement learning (RL) is widely recognized as the cornerstone for eliciting long chain-of-thought (CoT) reasoning, self-reflection, and mathematical and algorithmic problem-solving. However, a prevailing consensus in the community centers on the so-called "alignment tax": models trained with aggressive human preference alignment or RL objectives frequently exhibit degraded factual recall across knowledge-intensive benchmarks. Prior literature has predominantly interpreted this degradation as a fundamental tradeoff between logical reasoning and parametric memory storage, asserting that post-training overwrites or erodes stored factual weights.
Nevertheless, prior investigations on the alignment tax have largely focused on flat factual question answering and coarse instruction tuning, overlooking the rich structural topology of parametric memory. Real-world world knowledge is rarely organized as isolated, unstructured tuples; rather, it is embedded within relational networks, taxonomies, and formal ontologies. In specialized domains such as clinical coding (e.g., ICD-9/10, ATC) or patent classification (e.g., IPC), determining a precise leaf concept requires searching across an enormous discrete code space. When standard instruction-tuned models perform single-hop direct recall, they often fail or hallucinate due to the sheer size of the candidate pool. In contrast, reasoning models frequently demonstrate superior accuracy on these tasks. Whether this superiority stems from newly acquired knowledge or from systematic hierarchical navigation mimicking human expert traversal remains an unresolved question.
To disentangle knowledge acquisition from knowledge retrieval paths, this paper challenges the conventional wisdom that RL sacrifices memorization for reasoning. The authors hypothesize that instruct models already possess the underlying parametric knowledge, but lack the procedural navigation policy required to surface it, whereas RL implicitly internalizes structured hierarchical traversal over memory. Inspired by prompt optimization paradigms that bridge post-training gaps, the authors design structured prompting templates that explicitly guide instruct models through taxonomic levels. Strikingly, structured prompting recovers up to 65%–100% of the accuracy gap between instruct and reasoning models across five major model families, demonstrating that the missing knowledge is latent rather than absent. The core idea is to reframe the instruct–reasoning performance gap on knowledge-intensive tasks as an access and traversal bottleneck rather than a knowledge deficit, and to demonstrate via relation-disjoint controlled injection and layerwise representational probes that RL with iterated on-policy exploration unlocks latent parametric knowledge by reshaping question-navigation pathways while preserving base factual representations.
Method¶
Overall Architecture¶
The investigation is structured around three empirical pillars: diagnostic behavioral evaluation across diverse knowledge topologies, controlled causal isolation of RL via relation-disjoint knowledge injection, and internal mechanistic analysis. First, the authors evaluate eight instruct-reasoning model pairs across relational (PopQA), taxonomic (IPC-Lookup), and ontological (MedConceptsQA) benchmarks under direct answering, free-form chain-of-thought, and structured hierarchical prompting. Second, to eliminate confounding effects from pretraining data exposure, they formulate a controlled knowledge injection setup using Infini-gram n-gram counts and relation-disjoint training/test splits. Third, the authors introduce depth-stratified nearest common ancestor benchmarks (IPC-CAR and DBO-CAR) with a path-matching metric, accompanied by 64-layer representation probes and distillation failure-mode categorization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Parametric Knowledge Input<br/>PopQA / IPC / MedConceptsQA"] --> B["Structured Prompt Navigation<br/>Explicitly eliciting latent hierarchical traversal"]
B --> C["Relation-Disjoint Knowledge Injection<br/>Decoupling training exposure from retrieval access"]
C --> D["Iterated On-Policy Exploration Analysis<br/>Decomposing rollout freshness in GRPO and RFT"]
D --> E["Depth-Stratified Common Ancestor Retrieval<br/>IPC-CAR / DBO-CAR with PathMatch evaluation"]
E --> F["Layerwise Cosine Representation & Distillation Probes<br/>Query reshaping vs. surface self-correction failure"]
Key Designs¶
1. Structured Prompt Navigation: Decoupling Knowledge Storage from Retrieval Access Direct QA and free-form CoT prompting permit models to either guess an immediate answer or generate unstructured justifications, which obscures whether an incorrect output is caused by absent knowledge or an inefficient retrieval policy. To diagnose latent parametric knowledge, the authors design Structured Prompting. For multiple-choice datasets with explicit taxonomies (IPC-Lookup and MedConceptsQA), the prompt requires models to follow an explicit three-step procedure: first, recall the general category, chapter, and structural breakdown of the target code; second, analyze and eliminate distractors based on their structural locations in the taxonomy; third, select and justify the final choice. For open-ended relational QA (PopQA), the prompt directs the model to identify the entity-relation type, recall intermediate contextual anchor facts, and then synthesize the answer. When instruct models are provided with this explicit traversal scaffold, their performance jumps significantly toward that of reasoning models, confirming that the factual knowledge was already encoded in model weights.
2. Relation-Disjoint Knowledge Injection: Isolating RL Access Gains from Data Exposure To definitively counter the counterargument that RL improves factual recall simply by exposing models to new facts or overlapping relational patterns, the authors design a rigorous controlled injection experiment on PopQA. Using Infini-gram over the massive Dolma pretraining corpus, they measure the exact token co-occurrence frequency of subject entity-answer pairs. They extract a low-frequency, non-extractable training set (\(a=0\) pretraining co-occurrences, where Qwen3-8B/30B-Instruct fail across 3 independent runs) consisting of 731 samples, and a high-frequency, non-extractable held-out test set (\(a > 100\) co-occurrences, yet persistently inaccessible to the base instruct models) consisting of 208 samples. Crucially, the training and test sets are partitioned into mutually disjoint relational categories (e.g., training relations cover director, screenwriter, genre, producer, author; test relations cover composer, father, capital, place of birth, occupation). Models are trained with RL exclusively on the rare facts until convergence and tested on the held-out frequent facts, ensuring that any performance gain cannot be attributed to data memorization or relational transfer.
3. Iterated On-Policy Exploration Analysis: Identifying the Core Algorithmic Driver To dissect which component of the RL recipe facilitates parametric knowledge access, the authors decompose the training objective along three axes: self-generated versus external supervision, sample refresh frequency, and contrastive gradient penalization. Holding the training data (731 rare facts) and hyperparameters constant on Qwen3-8B-Instruct, they benchmark six objectives: Standard SFT (training on target answers), Static Rejection-Sampling Fine-Tuning (Static RFT, where rollouts are sampled once from the initial policy), Iterated RFT (where rollouts are regenerated from the current policy per batch), and Group Relative Policy Optimization (GRPO) under three refresh frequencies—Offline GRPO (static rollouts), Epoch-refresh GRPO (regenerated once per epoch), and Online GRPO (fresh rollouts generated before every gradient step). Using paired bootstrap confidence intervals, they quantify the marginal contribution of rollout freshness versus contrastive advantages.
4. Depth-Stratified Common Ancestor Retrieval: Evaluating Traversal Fidelity Across Hierarchy Depths To verify whether reasoning models truly internalize a scalable tree-search traversal rather than shallow heuristics, the authors construct two depth-stratified datasets: IPC-CAR (International Patent Classification) and DBO-CAR (DBpedia Ontology Common Ancestor Retrieval). Each task provides two leaf concept nodes and requires identifying their Nearest Common Ancestor (NCA) across controlled retrieval depths ranging from 2 to 6 edges. To evaluate traversal fidelity, the authors formulate the Path Matching Score (\(\text{PathMatch}\)), which balances set-level retrieval accuracy with topological sequence order: $\(\text{PathMatch} = \frac{\text{F1} + \text{CSS}}{2}\)$ $\(\text{CSS} = \frac{|\text{LCS}(\text{predicted}, \text{ground\_truth})|}{|\text{ground\_truth\_ancestors}|}\)$ where F1 is the harmonic mean of precision and recall over predicted versus ground-truth ancestors, and CSS measures the Longest Common Subsequence normalized by path length. This metric penalizes node hallucinations, omissions, and out-of-order traversal hops.
5. Layerwise Cosine Representation & Distillation Failure-Mode Probes: Question Reshaping vs. Surface-Level Mimicry To trace internal mechanistic shifts across the network, the authors extract the final-token hidden state \(h_\ell \in \mathbb{R}^d\) across all 64 layers of Qwen2.5-32B and Mistral-Small-24B variants (Base, Instruct, Reasoning, Distilled) for 100 matched pairs of questions (\(Q\)) and declarative ground-truth answers (\(A\)) from MedConceptsQA. Cosine similarity is computed both inter-model (comparing \(Q_{\text{base}}\) to \(Q_{\text{post-trained}}\) and \(A_{\text{base}}\) to \(A_{\text{post-trained}}\)) and intra-model (\(Q\) vs \(A\) within each variant): $\(\cos_{(a,b)}(\ell) = \frac{1}{N} \sum_{i=1}^N \frac{h_\ell^{(a)}(i)^\top h_\ell^{(b)}(i)}{\|h_\ell^{(a)}(i)\|_2 \|h_\ell^{(b)}(i)\|_2}\)$ In addition, to explain why SFT-distilled models (such as DeepSeek-R1-Distill-Qwen2.5-32B) underperform both native reasoning and instruct models on parametric recall, the authors conduct a qualitative audit of 40 failure trajectories under structured prompting.
Loss & Training¶
In the controlled injection experiments, RL models are fine-tuned using GRPO with LoRA (rank 32) on Qwen3-8B-Instruct and Qwen3-30B-A3B-Instruct. Training is conducted with a per-device batch size of 16 and a group size of 16 rollouts per question for 3 epochs. Sampling temperature during rollout generation is set to 0.7, while evaluation uses greedy decoding (temperature 0.0). Optimizers use Adam (\(\beta_1=0.9, \beta_2=0.95, \epsilon=10^{-8}\)) with a constant learning rate schedule (\(4\times 10^{-5}\) for 8B and \(5\times 10^{-4}\) for 30B-A3B, selected via grid search on held-out validation queries). The reward function employs an exact string-containment check: 1.0 if any acceptable ground-truth synonym appears in the generated completion, and 0.0 otherwise. SFT baselines train on identical pairs using standard cross-entropy loss over target tokens.
Key Experimental Results¶
Main Results¶
The table below presents the majority-vote accuracy of eight Instruct and Reasoning model pairs across PopQA (relational), IPC-Lookup (taxonomic), and MedConceptsQA (ontological) under Direct QA, CoT, and Structured Prompting.
| Dataset | Model | Model Type | Direct QA | CoT | Structured | Gap Reduction (QA \(\to\) Struct.) |
|---|---|---|---|---|---|---|
| MedConceptsQA | Qwen2.5-32B | Instruct Reasoning \(\Delta\) |
0.379 0.482 +0.103 |
0.475 0.513 +0.038 |
0.469 0.505 +0.036 |
Gap reduced by 65.0% (0.103 \(\to\) 0.036) |
| MedConceptsQA | DeepSeek-V3 / R1 | Instruct Reasoning \(\Delta\) |
0.541 0.778 +0.237 |
0.632 0.790 +0.158 |
0.717 0.792 +0.075 |
Gap reduced by 68.4% (0.237 \(\to\) 0.075) |
| MedConceptsQA | DeepSeek-V3.1 | Instruct Reasoning \(\Delta\) |
0.526 0.794 +0.268 |
0.595 0.805 +0.210 |
0.785 0.774 -0.011 |
Instruct overtakes Reasoning |
| MedConceptsQA | Kimi-K2.5 | Instruct Reasoning \(\Delta\) |
0.699 0.905 +0.206 |
0.734 0.915 +0.181 |
0.875 0.911 +0.036 |
Gap reduced by 82.5% (0.206 \(\to\) 0.036) |
| PopQA | Qwen2.5-32B | Instruct Reasoning \(\Delta\) |
0.252 0.386 +0.134 |
0.288 0.380 +0.092 |
0.310 0.375 +0.065 |
Gap reduced by 51.5% (0.134 \(\to\) 0.065) |
| PopQA | DeepSeek-V3 / R1 | Instruct Reasoning \(\Delta\) |
0.430 0.513 +0.083 |
0.505 0.557 +0.052 |
0.569 0.569 +0.000 |
Gap completely closed |
| IPC-Lookup | DeepSeek-V3 / R1 | Instruct Reasoning \(\Delta\) |
0.415 0.638 +0.223 |
0.596 0.670 +0.074 |
0.553 0.574 +0.021 |
Gap reduced by 90.6% (0.223 \(\to\) 0.021) |
| IPC-Lookup | Qwen3-30B-A3B | Instruct Reasoning \(\Delta\) |
0.309 0.426 +0.117 |
0.319 0.372 +0.053 |
0.436 0.457 +0.021 |
Gap reduced by 82.1% (0.117 \(\to\) 0.021) |
Ablation Study: Decomposition of Training Objectives¶
On Qwen3-8B-Instruct, the controlled knowledge injection task trains on 731 rare, non-extractable facts and evaluates recall on 208 relation-disjoint, highly frequent but previously inaccessible facts. The ablation results isolating rollout source, refresh frequency, and loss mechanics are shown below:
| Training Objective | Rollout & Refresh Mechanism | Held-Out Acc. (%) | Pairwise \(\Delta\) Acc. (pp) | 95% Paired-Bootstrap CI | Note |
|---|---|---|---|---|---|
| Zero-Shot Base | Untrained on target tasks | 0.00% | - | - | Complete failure on inaccessible set |
| Standard SFT | External ground-truth targets | 0.96% | Baseline | - | Minimal transfer across disjoint relations |
| Static RFT | Rollouts sampled once from policy \(\pi_0\) | 2.40% | +1.44 vs SFT | [-0.96, +3.85] pp | Self-generation yields slight, non-significant gain |
| Offline GRPO (No-refresh) | Rollouts sampled once from \(\pi_0\), group normalized | 3.37% | +0.97 vs Static RFT | - | Contrastive advantage without policy refresh |
| Epoch-Refresh GRPO | Rollouts refreshed once per epoch | 7.69% | +4.33 vs No-refresh | [+0.48, +8.65] pp | Statistically significant jump with periodic refresh |
| Iterated RFT | Rollouts refreshed every batch from \(\pi_t\) | 9.62% | +7.21 vs Static RFT | [+3.37, +11.54] pp | Iterated on-policy sampling drives the majority of gain |
| Online GRPO | Rollouts refreshed online every gradient step | 10.10% | +0.48 vs Iterated RFT +6.73 vs No-refresh |
[-3.85, +4.81] pp [+2.88, +11.06] pp |
Online GRPO matches Iterated RFT (CI spans 0) |
Key Findings¶
- Structured Traversal Re-surfaces Latent Memories: Structured prompting dramatically closes the instruct–reasoning gap across all evaluated models. On MedConceptsQA, Kimi-K2.5-Instruct drops its "All Incorrect" rate from 29% to 6%, while "All Correct" answers rise from 60% to 78% (matching Kimi-K2.5-Think's 82%). This response redistribution confirms that the underlying information was already encoded during pretraining and merely required effective hierarchical navigation to be elicited.
- Iterated Policy Refresh Trumps Contrastive Penalization: In the objective decomposition experiments, transitioning from external supervision (SFT: 0.96%) to static self-rollouts (Static RFT: 2.40%) provides only marginal benefit. However, incorporating iterated on-policy rollout refresh yields massive performance gains (Iterated RFT: 9.62%, Online GRPO: 10.10%). Crucially, the difference between Online GRPO and Iterated RFT (+0.48 pp, 95% CI [-3.85, +4.81]) is statistically indistinguishable from zero, proving that group-relative contrastive penalties contribute negligibly compared to the continuous exploration driven by on-policy policy updates.
- Reasoning Traversal Advantage Scales with Retrieval Depth: On IPC-CAR, while DeepSeek-V3 outperforms DeepSeek-R1 at depth 2 (PathMatch 0.865 vs 0.725), R1 overtakes V3 at depth 4 and widens its lead to +0.107 at depth 6 (0.434 vs 0.327). A parallel trend is observed on DBO-CAR (gap expands from -0.083 at depth 2 to +0.054 at depth 6). This divergence demonstrates that reasoning models learn a scalable procedural traversal capability that excels specifically as search complexity increases.
- Distillation Traps Models in "Surface-Level Self-Correction": Distilling DeepSeek-R1 reasoning traces into Qwen2.5-32B via SFT yields inferior recall compared to the native instruct and reasoning models (MedConceptsQA: 0.375 for Distilled vs. 0.379 Instruct and 0.482 Reasoning). Qualitative audit reveals that 80% of distillation errors exhibit superficial linguistic self-correction: models emit markers like "Wait..." or "Alternatively...", but remain anchored to their initial incorrect hypotheses, failing to uncommit or explore alternative taxonomic branches.
Highlights & Insights¶
- Reframing the "Alignment Tax" Paradigm: Rather than accepting factual degradation as an unavoidable consequence of post-training alignment, this study reveals that alignment primarily harms flat direct recall. For structured, multi-level knowledge, RL post-training actually enhances retrieval by internalizing tree-search navigation.
- Causal Isolation via Relation-Disjoint Injection: Combining Dolma-wide Infini-gram n-gram statistics with strictly orthogonal training and test relations represents an exemplary methodological standard for disentangling parametric knowledge accessibility from spurious training data exposure.
- Mechanistic Evidence from Internal Representations: Layerwise cosine similarity analysis reveals that post-training leaves declarative factual representations nearly identical to the base model (mean layerwise similarity of 0.961–0.976), whereas query representations diverge dramatically (similarity falling to 0.808–0.950). This confirms mathematically that RL restructures how questions are interpreted and routed rather than altering stored factual memories.
Limitations & Future Work¶
- N-Gram Co-occurrence as Exposure Proxy: Pretraining co-occurrence counts derived from Dolma serve as a statistical proxy for exposure, but do not capture complex higher-order associations or synthetic variations present in private corporate pretraining corpora.
- Commercial Pretraining Recipe Opacity: While controlled injection isolates RL mechanisms on open-weight base models, comparisons between frontier commercial checkpoints (e.g., DeepSeek-V3 vs R1) may be influenced by undisclosed data filtering, curriculum mixtures, and SFT distillation stages.
- Future Directions: Future post-training research should explore "traversal-optimized RL," training models explicitly to discover and navigate multi-hop knowledge graphs, rather than treating knowledge acquisition as a brute-force memorization task.
Related Work & Insights¶
- vs Gekhman et al. (2026) "Thinking to Recall": While Gekhman et al. study single-hop factuality and attribute recall gains to computational buffering and factual priming under pass@k metrics, this work evaluates reliable majority-vote recall over explicit hierarchical taxonomies (IPC, MedConceptsQA) and uncovers the primacy of iterated on-policy exploration.
- vs Lin et al. (2024) & Sorensen et al. (2026) on Alignment Tax: Prior studies established that RLHF reward scaling monotonically degrades factual accuracy; this paper demonstrates that structured taxonomic navigation represents an exception where RL substantially improves factual access over instruction tuning.
- vs SFT-based CoT Distillation (Guo et al., 2025; Yue et al., 2025): While distillation matches or exceeds RL on mathematical reasoning, this paper discovers that distillation fails on knowledge retrieval because supervised imitation lacks the trial-and-error exploration necessary to abandon erroneous retrieval hypotheses.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering reframing of the alignment tax; conceptualizes reasoning-aided recall as hierarchical traversal over latent memory]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluation across 5 model families, relation-disjoint controlled injection, 6 objective ablations, depth stratification, and 64-layer internal probes]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally cohesive narrative, rigorous causal methodology, clear conceptual definitions, and compelling visual empirical figures]
- Value: ⭐⭐⭐⭐⭐ [Highly impactful for LLM post-training, knowledge retrieval architectures, CoT distillation practices, and understanding the geometry of parametric memory]