Skip to content

Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Safety
Keywords: Reinforcement Learning, Parametric Knowledge Retrieval, Hierarchical Traversal, Alignment Tax, Structured Prompting

TL;DR

Challenging the prevailing assumption that reinforcement learning degrades factual recall through an alignment tax, this paper reveals that Instruct models suffer primarily from a knowledge access and navigation deficit rather than memory loss, and demonstrates that RL internalizes structured hierarchical traversal over latent parametric memory driven by iterated on-policy exploration.

Background & Motivation

As large language models (LLMs) accumulate vast stores of parametric knowledge during pre-training, subsequent post-training stagesโ€”including supervised fine-tuning (SFT), distillation, and reinforcement learning (RL)โ€”are standardly deployed to steer models toward instruction following and multi-step complex reasoning. However, an extensive body of recent literature points to an apparent "alignment tax", wherein optimizing models for reasoning or conversational alignment coincides with a precipitous drop in factual recall on knowledge-intensive benchmarks. Conventional wisdom posits a zero-sum trade-off: reasoning-focused RL encourages procedural problem-solving by sacrificing the fidelity and capacity of static parametric memory.

Nevertheless, prior empirical evaluations of the alignment tax have largely focused on flat, single-hop factual associations or closed-box commercial post-training recipes. This leaves open a fundamental theoretical ambiguity: does post-training actively purge or overwrite stored facts, or does it merely leave models unable to effectively extract and navigate their existing representations? In reality, knowledge spans a continuum of organizationโ€”from unstructured entity-relation associations to deeply nested, formal taxonomies and ontologies such as the International Patent Classification (IPC) and clinical ontologies (ICD-9/10, ATC). In such structured domains, direct single-step recall inevitably falters in vast combinatorial search spaces, whereas a systematic top-down search could reliably locate the target concept.

This paper cuts directly to the core of this dichotomy. Evaluating eight instructโ€“reasoning model pairs across five premier model families, the authors uncover that reasoning models actually outshine their instruct counterparts by 13 to 24 percentage points on factual recall benchmarks. Crucially, explicitly prompting Instruct models to perform hierarchical traversal recovers most of this performance deficit, proving that the underlying knowledge remains latent rather than erased. Core idea: Reinforcement learning does not inject new factual information or degrade parametric memory; rather, it internalizes systematic hierarchical traversal over latent pre-trained knowledge, primarily driven by iterated on-policy exploration.

Method

Overall Architecture

The paper formulates an experimental and diagnostic pipeline to cleanly separate factual storage from factual navigation across four progressive investigations. First, it probes eight instructโ€“reasoning model pairs across three benchmark domains (relational PopQA, taxonomic IPC-Lookup, ontological MedConceptsQA) under Direct QA, standard Chain-of-Thought (CoT), and Structured Prompting to verify whether the performance gap can be closed via external navigation instructions. Second, it designs a controlled knowledge-injection protocol utilizing pre-training corpus n-gram indexing (Infini-gram over Dolma) to isolate RL's navigation benefits from data exposure confounds using strictly relation-disjoint unseen splits. Third, it introduces depth-stratified common ancestor retrieval benchmarks (IPC-CAR and DBO-CAR) with a custom path matching metric to analyze how traversal scales with retrieval complexity. Finally, it conducts layerwise hidden representation probing and qualitative distillation failure analysis to uncover the mechanistic roots of learned navigation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query<br/>(Factual, Patent, or Medical Code)"] --> B["Structured Prompting Traversal<br/>Explicit Hierarchical Decomposition"]
    B --> C["Controlled Knowledge Injection<br/>Zero Exposure Cross-Relation Generalization"]
    C --> D["Iterated On-Policy Exploration<br/>Dynamic Rollout Policy Updating"]
    D --> E["Deep Hierarchical Navigation<br/>NCA Traversal & Path Matching"]
    E --> F["Exact Factual Output with Preserved Core Latent Knowledge"]

Key Designs

1. Structured Prompting Traversal: Externalizing hierarchical navigation to unlock latent knowledge

To test whether the instructโ€“reasoning gap is an access bottleneck rather than absent knowledge, the authors devise Structured Prompting. Unlike Direct QA (which demands only the final answer) and standard CoT (which generates unconstrained rationales), Structured Prompting instructs the model to explicitly recall the high-level category and structural breakdown of a code or concept before inspecting candidate options and pruning distractors. On MedConceptsQA, structured prompting shrinks the 23.7 pp gap between DeepSeek-V3 and DeepSeek-R1 down to 7.5 pp. Response distribution analysis shows that for Kimi-K2.5-Instruct, transitioning from Direct QA to Structured Prompting reduces "All Incorrect (0/3)" responses from 29% to 6%, while elevating "All Correct (3/3)" responses from 60% to 78%. Meanwhile, reasoning models exhibit negligible shifts because their internalized navigation policy already operates near ceiling, confirming that the factual knowledge was present all along inside Instruct models.

2. Controlled Knowledge Injection: Disentangling navigation gains from data exposure

To definitively rule out the hypothesis that RL's factual recall gains stem from data memorization during fine-tuning, the authors construct a frequency-stratified, relation-disjoint controlled injection experiment using PopQA. Utilizing Infini-gram on the Dolma pre-training corpus, they extract entity-answer co-occurrence counts \(a\). They construct a training set (\(n=731\)) of unseen, non-extractable facts (\(a=0\), zero model accuracy) and a held-out evaluation set (\(n=208\)) of high-exposure, inaccessible facts (\(a > 100\), zero baseline accuracy). The two sets enforce disjoint relation types (e.g., training on director/screenwriter/genre, evaluating on composer/father/capital). Applying GRPO exclusively on the rare unseen set improves recall on the held-out frequent set from 0% to 10.10% on Qwen3-8B-Instruct (compared to a mere 0.96% for SFT on identical data). Because the held-out facts were never seen during post-training and share no relational overlap, this demonstrates that RL cultivates a generalized parametric extraction policy.

3. Iterated On-Policy Exploration: Pinpointing the algorithmic engine of knowledge access

To determine which algorithmic facet of RL drives factual navigation, the authors decompose the training objective along three axes: self-generated vs. external targets, rollout refresh frequency, and contrastive advantage weighting. Holding data and base architecture constant on Qwen3-8B-Instruct, they benchmark SFT, Static Rejection-sampling Fine-Tuning (Static RFT), Iterated RFT, Offline GRPO, Epoch-refresh GRPO, and Online GRPO. The results demonstrate that replacing external targets with self-generated samples provides only modest gains (Static RFT achieves 2.40%), whereas refreshing rollouts iteratively across training batches produces the overwhelming majority of improvement (Iterated RFT reaches 9.62%). Furthermore, contrastive advantage weighting adds no statistically significant gain once on-policy iteration is in place (Online GRPO achieves 10.10%, \(\Delta = +0.48\) pp over Iterated RFT with bootstrap CI \([-3.85, 4.81]\)). Thus, continuous on-policy trial-and-error exploration is the decisive mechanism enabling models to discover latent retrieval pathways.

4. Deep Hierarchical Navigation: Stratified ancestor retrieval and path matching fidelity

To investigate how navigation functions as retrieval complexity increases, the authors introduce two depth-stratified Nearest Common Ancestor (NCA) benchmarks: IPC-CAR (derived from WIPO patent classifications) and DBO-CAR (derived from DBpedia ontologies) for traversal depths \(D \in [2, 6]\). To measure both ancestral set recovery and step order fidelity, they introduce the Path Matching Score:

\[\text{PathMatch} = \frac{\text{F1} + \text{CSS}}{2}, \quad \text{CSS} = \frac{|\text{LCS}(\text{predicted}, \text{ground\_truth})|}{|\text{ground\_truth ancestors}|}\]

where \(\text{F1}\) computes precision and recall over ancestor sets, and \(\text{CSS}\) evaluates the normalized Longest Common Subsequence against ground truth. On IPC-CAR, a striking crossover occurs: while DeepSeek-V3 leads at shallow depth 2 (PathMatch 0.865 vs. 0.725 for DeepSeek-R1), DeepSeek-R1 decisively overtakes V3 from depth 4 onwards, widening its margin to +0.107 at depth 6. This confirms that reasoning models possess an intrinsic traversal capability that scales robustly with topological retrieval depth.

Loss & Training

The controlled injection setup implements GRPO using LoRA adaptation (rank 32). Models are optimized using Adam (\(\beta_1=0.9, \beta_2=0.95, \epsilon=10^{-8}\)) with constant learning rates (\(4\times 10^{-5}\) for Qwen3-8B and \(5\times 10^{-4}\) for Qwen3-30B-A3B) across 3 epochs, a batch size of 16, and a group rollout size of 16. Advantages are normalized per question group against a graded string-matching reward (\(R=1.0\) if any valid answer synonym is retrieved, \(R=0.0\) otherwise).

Key Experimental Results

Main Results

The primary evaluation evaluates eight model pairs across relational (PopQA), taxonomic (IPC-Lookup), and ontological (MedConceptsQA) benchmarks under Direct QA, CoT, and Structured Prompting reporting Majority Vote Accuracy:

Model Family Model Specs Variant PopQA (QA / CoT / Struct.) IPC-Lookup (QA / CoT / Struct.) MedConceptsQA (QA / CoT / Struct.)
Qwen2.5 32B Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.252 / 0.288 / 0.310
0.386 / 0.380 / 0.375
+0.134 / +0.092 / +0.065
0.300 / 0.370 / 0.400
0.350 / 0.360 / 0.360
+0.050 / -0.010 / -0.040
0.379 / 0.475 / 0.469
0.482 / 0.513 / 0.505
+0.103 / +0.038 / +0.036
Qwen3 8B Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.205 / 0.252 / 0.262
0.237 / 0.267 / 0.262
+0.032 / +0.015 / +0.000
0.266 / 0.277 / 0.372
0.298 / 0.394 / 0.394
+0.032 / +0.117 / +0.022
0.338 / 0.337 / 0.406
0.400 / 0.424 / 0.421
+0.062 / +0.087 / +0.015
Qwen3 30B-A3B Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.231 / 0.279 / 0.302
0.288 / 0.317 / 0.312
+0.058 / +0.037 / +0.010
0.309 / 0.319 / 0.436
0.426 / 0.372 / 0.457
+0.117 / +0.053 / +0.021
0.398 / 0.404 / 0.489
0.498 / 0.506 / 0.476
+0.100 / +0.102 / -0.013
Qwen3 235B-A22B Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.429 / 0.491 / 0.541
0.494 / 0.562 / 0.567
+0.065 / +0.071 / +0.026
0.372 / 0.383 / 0.596
0.457 / 0.404 / 0.574
+0.085 / +0.021 / -0.022
0.542 / 0.548 / 0.631
0.641 / 0.656 / 0.580
+0.099 / +0.108 / -0.051
DeepSeek V3 / R1 Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.430 / 0.505 / 0.569
0.513 / 0.557 / 0.569
+0.083 / +0.052 / +0.000
0.415 / 0.596 / 0.553
0.638 / 0.670 / 0.574
+0.223 / +0.074 / +0.021
0.541 / 0.632 / 0.717
0.778 / 0.790 / 0.792
+0.237 / +0.158 / +0.075
DeepSeek V3.1 / R1 Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.429 / 0.506 / 0.565
0.429 / 0.508 / 0.567
+0.003 / +0.002 / +0.002
0.468 / 0.500 / 0.606
0.479 / 0.511 / 0.585
+0.011 / +0.011 / -0.021
0.526 / 0.595 / 0.785
0.794 / 0.805 / 0.774
+0.268 / +0.210 / -0.011
Kimi K2 Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.507 / 0.533 / 0.566
0.580 / 0.662 / 0.652
+0.073 / +0.129 / +0.086
0.511 / 0.521 / 0.543
0.575 / 0.585 / 0.564
+0.064 / +0.064 / +0.021
0.595 / 0.634 / 0.750
0.834 / 0.829 / 0.830
+0.239 / +0.195 / +0.080
Kimi K2.5 Instruct
Reasoning
\(\Delta\) (Reasoning Gap)
0.507 / 0.573 / 0.649
0.597 / 0.700 / 0.682
+0.100 / +0.127 / +0.033
0.521 / 0.500 / 0.553
0.574 / 0.574 / 0.585
+0.053 / +0.074 / +0.032
0.699 / 0.734 / 0.875
0.905 / 0.915 / 0.911
+0.206 / +0.181 / +0.036

Ablation Study & In-Depth Analysis

1. Decomposition of Post-Training Objectives on Held-out PopQA Facts (Qwen3-8B-Instruct, \(n=208\))

Post-Training Algorithm / Objective Held-Out Test Acc. (%) 95% Wilson CI Pairwise Difference (\(\Delta\) Acc / 95% Paired-Bootstrap CI)
Zero-shot Baseline 0.00% โ€” โ€”
Supervised Fine-Tuning (SFT) 0.96% [0.26, 3.42] Baseline
Static Rejection-sampling Fine-Tuning (Static RFT) 2.40% [1.03, 5.48] Static RFT โˆ’ SFT: +1.44 pp ([-0.96, 3.85])
Offline GRPO (No-refresh) 3.37% [1.64, 6.78] Fixed initial rollouts
Epoch-refresh GRPO 7.69% [4.84, 11.99] Epoch โˆ’ No-refresh: +4.33 pp ([0.48, 8.65])
Iterated RFT (Per-batch dynamic refresh) 9.62% [6.37, 14.28] Iterated RFT โˆ’ Static RFT: +7.21 pp ([3.37, 11.54])
Online GRPO (Fresh rollout per update) 10.10% [6.76, 14.83] Online GRPO โˆ’ Iterated RFT: +0.48 pp ([-3.85, 4.81])

2. Depth-Stratified Hierarchy Traversal: DeepSeek-V3 vs DeepSeek-R1 on IPC-CAR (\(n=20\) per depth)

Retrieval Depth (\(D\)) V3 Final Acc. R1 Final Acc. \(\Delta\) Acc (R1 - V3) V3 PathMatch R1 PathMatch \(\Delta\) PathMatch (R1 - V3)
Depth 2 0.900 0.800 -0.100 0.865 0.725 -0.140
Depth 3 0.850 0.700 -0.150 0.576 0.549 -0.027
Depth 4 0.800 0.700 -0.100 0.454 0.473 +0.019
Depth 5 0.750 0.800 +0.050 0.416 0.465 +0.049
Depth 6 0.750 0.800 +0.050 0.327 0.434 +0.107

3. Layerwise Hidden Representation Cosine Similarity (Qwen2.5-32B vs Variants on ICD9PROC)

Representation Comparison Variant Early Layers (1โ€“21) Mid Layers (22โ€“43) Late Layers (44โ€“64) Overall Mean Min Similarity Min Layer
Question Cosine Sim (\(Q_{\text{base}} \leftrightarrow Q_{\text{post}}\)) Instruct
Reasoning
Distilled
0.990
0.982
0.911
0.970
0.942
0.718
0.938
0.923
0.789
0.966
0.950
0.808
0.729
0.810
0.634
Layer 64
Layer 64
Layer 36
Answer Declarative Sim (\(A_{\text{base}} \leftrightarrow A_{\text{post}}\)) Instruct
Reasoning
Distilled
0.994
0.987
0.955
0.976
0.951
0.864
0.958
0.944
0.854
0.976
0.961
0.892
0.871
0.876
0.778
Layer 64
Layer 64
Layer 64

Key Findings

  • Structured Traversal Resolves the Retrieval Gap: Guiding Instruct models along hierarchical paths substantially eliminates the accuracy advantage of reasoning models, with Instruct variants matching or surpassing reasoning models on several benchmarks (e.g., DeepSeek-V3.1 on MedConceptsQA reaching 78.5% vs 77.4%).
  • On-Policy Exploration Outweighs Contrastive Loss: Iterating the sampling policy accounts for almost all RL gains (Iterated RFT reaches 9.62% vs 2.40% for Static RFT). Adding GRPO's contrastive gradient advantage over below-average rollouts contributes negligibly (+0.48 pp).
  • Depth-Dependent Advantage: The crossover in PathMatch scores on IPC-CAR demonstrates that reasoning models' specialized advantage is structural, expanding monotonically as hierarchical retrieval requires deeper traversals.
  • Query Processing Shift without Knowledge Degradation: Layerwise probing reveals that declarative factual representations remain remarkably unchanged after RL (\(A \leftrightarrow A\) similarity exceeds 0.96 across layers), whereas question representations diverge significantly, proving that RL optimizes question processing and navigation routing rather than altering memory content.
  • Superficial Self-Correction in Distilled Models: Error analysis of DeepSeek-R1-Distill-Qwen2.5-32B shows that 80% of distillation failures stem from linguistic self-correction mimicry (using tokens like "Wait..." without genuinely uncommitting from erroneous hypotheses), explaining why distillation fails to inherit RL's true exploration capacity.

Highlights & Insights

  • Reconceptualization of Alignment Tax: Reframes the apparent degradation of factual knowledge during RL as an access and traversal bottleneck rather than parametric forgetting.
  • Clean Causal Isolation: Deploys Infini-gram co-occurrence filtering across trillions of pre-training tokens paired with strict relation disjointness to isolate retrieval capability from data exposure artifacts.
  • Mechanistic Dissection of Distillation: Unveils that SFT-based CoT distillation produces superficial heuristic imitation rather than authentic trial-and-error state-space exploration.

Limitations & Future Work

  • Pre-training Exposure Proxy: While n-gram co-occurrence offers a rigorous frequency estimate, it remains an imperfect surrogate for complex semantic exposure during pre-training.
  • Opacity of Commercial Post-Training Recipes: Comparing publicly released instruct and reasoning models involves confounding factors due to proprietary multi-stage post-training mixtures.
  • Future post-training paradigms should explicitly train models on efficient parametric indexing and structured navigation primitives rather than solely expanding training token counts.
  • Contrasts with external RAG pipelines by proving that vast quantities of latent knowledge can be elicited natively from model weights via internalized traversal.
  • Bridges the gap with the Reversal Curse, showing that parametric knowledge accessibility is heavily directional and structurally conditioned.
  • Provides actionable guidance for agentic systems: zero-shot hierarchical prompting can recover deep reasoning accuracy from fast Instruct models without incurring heavy RL training overhead.

Rating

  • Novelty: โญโญโญโญโญ Decisively challenges the alignment tax narrative and formally formalizes hierarchical parametric traversal.
  • Experimental Thoroughness: โญโญโญโญโญ Extensive evaluations across five model families, controlled pre-training corpus filtering, and layerwise hidden state probes.
  • Writing Quality: โญโญโญโญโญ Rigorous conceptual structure, fluid narrative delivery, and meticulous analytical breakdowns.
  • Value: โญโญโญโญโญ Fundamental contributions to LLM post-training, factual retrieval, and hallucination mitigation.