Skip to content

Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/oneal2000/ReGrad
Area: LLM Reasoning
Keywords: continual post-training, knowledge injection, parametric memory, gradient retrieval, retrieval-augmented generation

TL;DR

Introduces REGRAD, a paradigm that treats document-induced gradients as retrievable knowledge units stored in an offline Gradient Bank, dynamically applying meta-learned gradients for query-specific, reversible parameter adaptation without cumulative weight drift or catastrophic forgetting.

Background & Motivation

Continual post-training (CPT) has established itself as the prevailing approach for internalizing emerging real-world knowledge into large language models after deployment. However, CPT fundamentally relies on repeatedly executing gradient updates over the same globally shared parameter space. As training sequences extend over time and corpora scale into millions of tokens, the network weights inexorably drift away from their pre-trained initialization. This cumulative parameter drift triggers severe catastrophic forgetting, progressively degrading foundational model capabilities such as complex reasoning, instruction following, and mathematical problem-solving. Furthermore, while an individual downstream user query only requires a narrow subset of domain facts, CPT indiscriminately accumulates updates from all unrelated documents into a single model state, hopelessly tangling relevant knowledge with millions of interfering parameter modifications.

Retrieval-augmented generation (RAG) sidesteps weight modification altogether by retrieving external text passages at inference time and prepending them to the input context. While this non-parametric strategy preserves the base model weights and allows reversible knowledge injection, it leaves the knowledge entirely outside the model's parameters. Theoretical and empirical studies show that deep procedural reasoning, associative memory, and latent concept manipulation operate fundamentally through parametric pathways rather than purely in-context attention. Moreover, concatenating long passages into prompts induces quadratic attention computation, increases token latency, and exposes the model to context distractions such as the "lost-in-the-middle" phenomenon and attention dilution. This exposes a fundamental trade-off: how can a system harvest the deep reasoning benefits of parametric knowledge internalization without accumulating irreversible weight drift?

This paper revisits the fundamental atomic unit of deep learning: the gradient. In standard training, gradients are treated as ephemeral optimization signals that are applied once to shift model weights and immediately discarded. In reality, a gradient encodes an exact, high-dimensional vector specifying how the model's parameters should shift to internalize the factual contents of a specific document. Core idea: decouple knowledge acquisition from permanent weight updates by pre-computing document-induced gradients offline as reusable "knowledge atoms" stored in an indexed Gradient Bank, dynamically retrieving query-relevant gradients at inference time for temporary parameter adaptation, and reverting to the untouched base model immediately after generation.

Method

Overall Architecture

REGRAD decouples persistent knowledge storage from transient parameter activation across three coordinated phases: Stage 1 performs bi-level meta-learning over low-rank adapter (LoRA) parameters to reshape unsupervised document reconstruction gradients into task-oriented adaptation directions; Stage 2 constructs an offline Gradient Bank by executing a single label-free backward pass over the target knowledge corpus using the meta-learned initialization; Stage 3 executes query-driven online inference, where Top-\(K\) relevant gradients are retrieved, linearly aggregated in the low-rank subspace, applied for one-step generation, and immediately reverted.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query q and External Knowledge Corpus D"] --> B["Stage 1: Bi-Level Meta-Learning for Gradient Shaping<br/>Inner unsupervised LM update + Outer context-free QA loss"]
    B --> C["Stage 2: Offline Gradient Bank Construction<br/>Low-rank backward pass storing document knowledge atoms"]
    C --> D["Stage 3: Retrieval-Augmented Parametric Adaptation<br/>Query retrieves Top-K gradients with linear tensor aggregation"]
    D --> E["Parametric Generation & Weight Rollback<br/>Generates answer without context and restores base weights"]

Key Designs

1. Bi-Level Meta-Learning for Gradient Shaping: Overcoming Surface Reconstruction Misalignment

Naive next-token prediction gradients are optimized strictly for verbatim surface-form memorization. Applying such raw optimization signals directly at test time introduces destructive parameter noise that degrades pre-trained reasoning abilities. REGRAD overcomes this via a MAML-style bi-level meta-learning objective that optimizes the adapter initialization \(\theta\) and per-tensor step sizes \(\alpha\). The inner loop computes an unsupervised language modeling gradient over a document passage \(C\):

\[\tilde{\mathcal{L}}_C(\theta) = -\frac{1}{T}\sum_{t=1}^T \log p_{(W,\theta)}(c_t \mid c_{<t}), \quad \theta_C = \theta - \alpha \odot \nabla_\theta \tilde{\mathcal{L}}_C(\theta)\]

Crucially, in the outer loop evaluation, the reference document \(C\) is completely excluded from the model prompt. The adapted model \(\theta_C\) must generate the correct answer \(a\) given only the query \(q\):

\[\min_{\theta, \alpha} \mathbb{E}_{(C,q,a) \sim \mathcal{D}_{\text{meta}}} \mathcal{L}_{\text{QA}}^C\left(\theta - \alpha \odot \sum_{k=1}^K g_{C_k}(\theta)\right)\]

Differentiating through the inner update forces \(\theta\) to act as a "gradient shaper," guaranteeing that label-free document gradients mathematically translate into effective parameter shifts for multi-hop reasoning and factual recall.

2. Compact Parametric Offline Gradient Bank: Eliminating Full-Model Storage and Runtime Backpropagation

Storing dense gradients across billions of base parameters \(W\) for millions of documents is storage-prohibitive. REGRAD restricts the trainable adaptation subspace to compact LoRA parameters \(\theta\) inserted into Transformer layers, while keeping the base LLM weights \(W\) completely frozen. Once optimal initialization \((\theta^*, \alpha^*)\) is meta-trained, the entire external corpus \(\mathcal{D}\) is processed offline: for each document chunk \(d_i\), a single backward pass computes its compact knowledge atom \(g_i = \nabla_\theta \tilde{\mathcal{L}}_{d_i}(\theta^*)\). With each atom requiring only \(2nr(h + l)\) parameters (where \(n\) is the number of adapted layers, \(r\) the rank, \(h\) the hidden dimension, and \(l\) the intermediate FFN size), storage is compressed to a few hundred kilobytes per document chunk, indexed alongside standard BM25 or dense retrieval keys.

3. Retrieval-Augmented Transient Weight Adaptation: Resolving Continual Learning Drift with Linear Additivity

During online inference, a user query \(q\) retrieves the Top-\(K\) relevant documents \(\{d_{i_k}\}_{k=1}^K\) and directly fetches their stored gradients \(\{g_{i_k}\}_{k=1}^K\). Because all stored knowledge atoms were derived from the identical meta-learned reference state \(\theta^*\), the linear additivity of gradients ensures that summing the retrieved vectors corresponds exactly to evaluating the gradient of their joint language modeling objective:

\[g(q) = \sum_{k=1}^K g_{i_k}, \quad \theta_q = \theta^* - \alpha^* \odot g(q)\]

This transient adaptation involves only elementary tensor additions and scalar multiplications taking 6–15 ms, completely bypassing expensive online backpropagation. The model executes forward generation under \(M(W, \theta_q)\) without inflating the prompt with retrieved text. Once the generation concludes, the weights are instantly rolled back to \(\theta^*\), completely severing the chain of cumulative parameter pollution across queries.

Loss & Training

In Stage 1, the outer optimization employs a meta-batch size of 16 for 1 epoch with bfloat16 mixed precision. Optimization utilizes a linear learning rate schedule with a peak learning rate of \(4\times 10^{-4}\), and per-tensor step sizes \(\alpha\) are initialized to \(1\times 10^{-2}\). Retrieval is evaluated in both a pure parametric mode (REGRAD) and a hybrid mode (REGRAD + ICL), where retrieved passages are simultaneously injected into both the context prompt and the parameter adaptation pathway.

Key Experimental Results

Main Results

Evaluations span nine benchmarks across three domains: General Wikipedia (2WQA, CWQ, HotpotQA), Legal (CaseHOLD, Learned Hands Family, HousingQA), and Biomedical (PubMedQA, MedQA, BioASQ) using LLaMA-3.2 (1B, 3B) and LLaMA-3.1 (8B) under the Top-1000 setting (Accuracy for closed-ended tasks, token-level F1 for open-ended QA):

Backbone Method 2WQA CWQ HotpotQA CaseHOLD LHF HousingQA PubMedQA MedQA BioASQ Overall Avg.
LLaMA-1B Direct Generation 21.27 31.71 16.08 42.1 41.1 64.3 17.3 6.24 42.6 31.41
Standard CPT 15.34 28.19 11.42 40.1 40.3 59.7 18.9 6.13 46.6 29.63
Instruction CPT 30.71 34.74 18.93 40.7 47.1 62.9 20.1 6.80 41.4 33.71
RAG (ICL) 24.08 33.58 24.01 42.7 41.2 65.8 70.2 7.88 71.2 42.29
PE-RAG 24.98 35.63 27.94 44.0 43.2 62.7 62.9 6.12 66.3 41.53
PRAG (Parametric RAG) 24.73 32.27 20.11 41.8 41.2 63.5 17.3 6.92 47.3 32.79
Fine-tuned-Direct 27.96 42.74 18.66 45.2 86.1 79.2 79.0 10.63 72.7 51.35
Fine-tuned-RAG 32.04 46.82 31.09 42.1 90.9 68.4 78.2 10.82 80.8 53.46
Fine-tuned-PRAG 31.43 45.26 24.73 65.1 89.5 81.9 92.4 10.78 76.8 57.54
REGRAD (Parametric Only) 34.61 47.18 30.72 65.7 94.8 82.1 93.3 10.12 74.6 59.24
REGRAD + ICL (Hybrid) 40.63 50.34 39.18 68.4 94.7 84.5 97.9 11.50 83.4 63.39
LLaMA-3B Direct Generation 16.58 36.69 18.77 55.8 46.1 47.7 88.1 10.63 77.8 44.24
Standard CPT 10.96 33.94 15.48 54.2 44.5 45.7 78.4 9.47 75.1 40.86
Instruction CPT 30.28 43.26 24.09 55.5 48.2 45.4 82.6 13.06 78.3 46.74
RAG (ICL) 20.13 29.84 24.11 63.2 42.2 51.2 82.8 7.13 74.2 43.87
PE-RAG 17.75 40.05 32.92 66.2 47.5 53.6 81.7 11.58 77.9 47.69
Fine-tuned-RAG 35.48 51.11 32.49 50.9 91.9 70.5 86.4 15.19 82.1 57.34
REGRAD (Parametric Only) 36.82 52.57 33.43 68.8 94.7 83.1 92.8 16.66 78.6 61.94
REGRAD + ICL (Hybrid) 44.18 54.08 43.47 71.3 94.6 84.8 97.9 18.21 85.7 66.03
LLaMA-8B Direct Generation 33.42 42.73 26.34 61.3 54.1 46.0 91.2 16.46 81.2 50.31
Standard CPT 25.67 40.24 23.51 58.1 53.8 46.2 90.3 15.59 80.2 48.18
Instruction CPT 36.31 46.58 27.44 63.8 54.0 49.5 38.7 19.24 60.6 44.02
Fine-tuned-RAG 37.17 54.21 39.16 59.6 91.9 69.5 92.7 23.22 84.6 61.34
Fine-tuned-PRAG 35.71 51.26 32.97 66.2 78.6 83.0 91.8 21.37 82.6 60.39
REGRAD (Parametric Only) 38.43 56.41 36.31 70.3 96.2 84.8 93.2 21.65 81.9 64.36
REGRAD + ICL (Hybrid) 45.06 57.52 45.27 74.1 94.8 85.4 98.1 19.83 85.3 67.26

Ablation Study

1. Continual Knowledge Injection vs. Weight Drift (LLaMA-8B, F1 Score)

Evaluating how sequential injection across a 10K document corpus impacts performance:

Injected Docs 2Wiki (CPT) 2Wiki (REGRAD) CWQ (CPT) CWQ (REGRAD) HotpotQA (CPT) HotpotQA (REGRAD) CPT Avg. REGRAD Avg.
2.5K Docs 27.14 33.95 43.32 55.91 21.73 23.26 30.73 37.71
5.0K Docs 28.26 34.12 44.55 57.16 21.43 27.65 31.41 39.64
7.5K Docs 27.43 36.12 45.72 59.84 22.46 31.84 31.87 42.60
10.0K Docs 20.25 37.70 45.89 60.10 17.26 34.35 27.80 44.05

2. Cross-Domain Transfer of Meta-Initialization (Macro F1 / Accuracy)

Evaluating whether a general-domain meta-initialization transfers to specialized verticals:

Model Scale Strategy General Domain Medical Domain Legal Domain
LLaMA-1B Base LLM 22.06 27.50 45.33
General-init REGRAD 42.05 28.12 43.33
Continued REGRAD 38.42 56.07 79.11
Mixed-domain REGRAD 42.45 55.35 76.44
LLaMA-3B Base LLM 25.36 59.72 48.22
General-init REGRAD 46.14 45.46 53.67
Continued REGRAD 43.18 61.72 79.44
Mixed-domain REGRAD 47.80 60.70 76.55

3. End-to-End Serving Latency (8×A100-80G, ms/Query)

Method Variant LLaMA-1B LLaMA-3B LLaMA-8B System Mechanism
Direct Generation 13.44 ms 18.36 ms 28.30 ms Pure forward pass without retrieval
RAG 17.03 ms 23.62 ms 34.63 ms Quadratic attention on Top-3 context
REGRAD Cached 16.89 ms 22.97 ms 30.84 ms Pre-computed gradient fetch & tensor addition
REGRAD + ICL 18.21 ms 26.60 ms 38.10 ms Hybrid parametric update + text context
REGRAD Online 32.04 ms 45.83 ms 56.31 ms Zero cache footprint with test-time backward pass

Key Findings

  • Essential Role of Meta-Learning: Directly using raw next-token prediction gradients without bi-level optimization causes catastrophic performance drops across all model scales, falling below Direct Generation. Raw gradients overfit surface tokens, introducing optimization noise; the meta-learning stage is indispensable for transforming raw gradients into structured adaptation signals.
  • Overcoming the CPT Forgetting Trap: As corpus scale increases to 10K documents, standard CPT degrades sharply due to accumulated weight drift (LLaMA-8B average F1 drops from 31.87 down to 27.80). Conversely, REGRAD scales monotonically to 44.05 F1 because a larger bank increases retrieval recall while every query applies only a clean, single-step update.
  • Synergy between Parametric and Contextual Knowledge: REGRAD + ICL achieves top-ranked performance across all three model tiers (reaching 67.26 Avg on LLaMA-8B), significantly outperforming both pure RAG and pure CPT variants. This demonstrates that internalizing knowledge into weights and supplying raw textual evidence operate along complementary cognitive pathways.
  • Gradient Interference at Higher Retrieval Depths: Varying \(k\) reveals an inverted-U trend peaking around \(k=3\). Beyond this threshold, irrelevant documents introduce gradient interference that degrades the temporary weight update, mirroring the attention dilution seen in long-context RAG.

Highlights & Insights

  • Reframes the gradient from an ephemeral optimization artifact into an indexed, permanent, and retrievable knowledge atom, decoupling knowledge integration from irreversible weight corruption.
  • Uses a bi-level meta-learning objective where the inner loop executes label-free language modeling and the outer loop enforces context-free QA, teaching the model to turn raw document text into task-adaptive parameter directions.
  • Seamlessly integrates with multi-LoRA serving runtimes (e.g., Punica, S-LoRA), enabling batched inference across heterogeneous user queries with negligible dynamic tensor-merging overhead (2 ms).

Limitations & Future Work

  • Dependence on Supervised QA Pairs during Meta-Training: The meta-learning phase requires QA pairs to supervise the outer loss; extending this to synthetic or self-supervised contrastive objectives would reduce data curation bottlenecks.
  • Storage Scaling for Web-Scale Corpora: Although low-rank gradients are compact (a few hundred KB per document chunk), scaling to billions of web documents demands progressive gradient quantization, sparsification, or hybrid tiering between cached and on-demand gradients.
  • Extension Beyond Factual Knowledge: The current evaluation focuses on factual QA; extending retrievable gradients to modular skill acquisition, agentic tool invocation, or dynamic safety policies remains an exciting frontier.
  • vs Continual Post-Training (CPT / Instruction CPT): Standard CPT permanently modifies shared parameters, inevitably triggering catastrophic forgetting of reasoning and math skills; REGRAD confines knowledge updates to query-specific transient states, leaving base weights permanently pristine.
  • vs Standard RAG / PE-RAG: In-context RAG expands prompt length and incurs quadratic computational complexity without parameterizing knowledge; REGRAD maintains standard \(O(|q|^2h)\) prompt complexity while unlocking complementary gains when combined with text retrieval.
  • vs Test-Time Training (TTT / Dynamic PRAG): Prior test-time fine-tuning methods require online backpropagation at inference time, adding hundreds of milliseconds of latency; REGRAD pre-computes gradients offline, reducing test-time update costs to microsecond tensor additions.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant conceptual redefinition of gradients as retrievable parametric knowledge units.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 3 model scales, 3 domains, 9 benchmarks, continual drift benchmarks, and end-to-end hardware latency.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical framing, intuitive narrative, and transparent ablation studies.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly promising path reconciling the lifelong learning capabilities of CPT with the safety and modularity of RAG.