Skip to content

Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/oneal2000/ReGrad
Area: LLM Reasoning
Keywords: Continual Post-Training, Retrieval-Augmented Generation, Retrievable Gradients, Bi-Level Meta-Learning, Catastrophic Forgetting

TL;DR

REGRAD treats document-specific parameter gradients as retrievable units of knowledge stored in an offline Gradient Bank, reshaping unsupervised causal language modeling updates into downstream QA adaptation directions via bi-level meta-learning to achieve query-specific parametric knowledge injection without cumulative weight drift.

Background & Motivation

Continual post-training (CPT) represents a predominant paradigm for enabling deployed large language models to internalize freshly emerging real-world facts directly into their parametric memory. By updating network weights across newly gathered corpora, CPT allows models to weave external knowledge into deep parametric pathways, thereby facilitating latent multi-hop deductions and query-guided factual abstraction. However, this parametric internalization comes at a severe structural expense: CPT continuously updates a single, globally shared parameter state. As sequential updates accumulate over extended temporal horizons or massive document streams, the model weights gradually drift away from their pre-trained manifold, inducing cumulative weight drift that triggers catastrophic forgetting of foundational reasoning capabilities and compromises general instruction-following proficiency. Furthermore, while an individual downstream user query demands only a localized sliver of newly introduced facts, standard CPT unconditionally blends modifications from thousands of unrelated documents into the exact same parameter space, contaminating task-specific representations with irrelevant optimization noise.

In stark contrast, retrieval-augmented generation (RAG) circumvents parameter modification altogether by dynamically retrieving pertinent text passages at test time and prepending them to the input context. While this non-parametric strategy preserves the base model weights intact and ensures clean reversibility, it remains constrained to in-context knowledge injection. The model must process external facts purely through its dynamic context window rather than drawing upon internalized parametric circuits. Empirical studies have demonstrated that an LLM's profound reasoning circuits and world knowledge are predominantly anchored within its internal weights; merely stuffing long passages into the context window frequently induces attention dilution, triggers lost-in-the-middle degradation, and increases quadratic inference latency without achieving genuine parametric comprehension. Hence, existing paradigms pose a fundamental dilemma: how can an intelligent system harvest the intrinsic benefits of deep parametric knowledge integration while entirely circumventing cumulative weight drift?

This paper resolves this tension by fundamentally rethinking the physical status of the gradient in post-training workflows. Conventional optimization treats gradients as transient, single-use directional derivatives that are immediately discarded once weight increments are applied. Yet from an information-theoretic angle, the gradient computed on an individual document precisely delineates the structural parametric deformation necessary for the network to internalize that specific textual knowledge. Instead of permanently and irreversibly accumulating every document gradient into a shared static checkpoint, one can decouple gradient computation from permanent model updates by pre-computing gradients offline, indexing them into an external Gradient Bank, and retrieving only query-relevant gradients for temporary inference-time adaptation. The core idea is to transform document-induced gradients into retrievable knowledge atoms via a bi-level meta-learning objective that reshapes unsupervised language-modeling gradients into generalizable downstream QA adaptation directions, enabling query-specific parametric injection without permanent weight drift.

Method

Overall Architecture

The REGRAD framework orchestrates a decoupled three-stage operational lifecycle across offline meta-optimization, corpus indexing, and online serving. In Stage 1 (Meta-Learning for Initialization), a bi-level optimization problem trains a compact LoRA adapter initialization \(\theta^*\) and layer-wise step-size vector \(\alpha^*\) so that an unsupervised next-token prediction gradient derived from an unlabeled document chunk directly translates into an optimal one-step update for answering downstream queries without external context. In Stage 2 (Gradient Bank Construction), using the fixed meta-learned parameters \((\theta^*, \alpha^*)\), REGRAD performs a single unsupervised forward and backward pass for each document chunk in the knowledge corpus, storing the resulting low-rank gradients alongside a text retrieval index as modular knowledge atoms. In Stage 3 (Online Inference), given an incoming query, standard retrieval fetches the Top-\(K\) relevant documents and their associated knowledge atoms, aggregates them via linear gradient summation, applies a temporary parameter-space update, generates the response with the base LLM, and instantaneously discards the temporary weights to fully restore the pristine base model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["User Query q"] --> B["Stage 1: Bi-Level Meta-Learning Shaper<br/>Optimizes LoRA init θ* and step sizes α*"]
    B --> C["Stage 2: Offline Gradient Bank Construction<br/>Stores unsupervised document gradients as knowledge atoms g_i"]
    C --> D["Stage 3: On-Demand Retrieval & Aggregation<br/>Linear sum of Top-K knowledge atoms into g(q)"]
    D --> E["Temporary Parameter Adaptation & Instant Rollback<br/>θ_q = θ* - α* ⊙ g(q) applied for inference, then reverted"]
    E --> F["Zero-Context Parametric Generation<br/>No retrieved text prepended to input prompt"]

Key Designs

1. Bi-Level Meta-Learning Shaper: Aligning Unsupervised Reconstruction Gradients with Downstream QA Reasoning

A standard causal language modeling gradient derived from next-token prediction merely reinforces verbatim surface-level token memorization and textual reconstruction rather than abstract factual retrieval or multi-hop query answering. Naively caching and applying raw language-modeling gradients injects noisy syntactic updates that corrupt the model's pre-trained attention patterns and degrade generation accuracy. REGRAD overcomes this misalignment by framing the initialization of the plug-in parameters as a bi-level meta-learning problem. The inner loop executes an unsupervised language-modeling step on unlabeled context chunks within the low-rank subspace, while the outer loop evaluates the updated model strictly on target QA pairs where the context passages are completely removed from the prompt. By backpropagating through the inner update trajectory, the meta-optimization reshapes the adapter parameters \(\theta\) into a specialized "gradient shaper" that forces unsupervised document gradients to function as task-oriented adaptation vectors.

2. Offline Gradient Bank Construction: Decoupling Document Ingestion into Compact Low-Rank Knowledge Atoms

Computing and indexing full-parameter gradients across millions of corpus documents would incur prohibitive storage costs and severe memory I/O bottlenecks during serving. REGRAD bounds the adaptation domain to a lightweight low-rank subspace by inserting LoRA adapters into designated transformer projection layers, freezing the massive underlying backbone weights \(W\). For every document chunk \(d_i\) in external corpus \(\mathcal{D}\), an unsupervised backward pass is executed at the frozen meta-initialization \(\theta^*\), yielding a compact parameter gradient \(g_i = \nabla_\theta \tilde{\mathcal{L}}_{d_i}(\theta^*)\) that acts as a discrete, retrievable knowledge atom. Each atom occupies only a few hundred kilobytes, and the entire construction cost is amortized once offline—requiring total compute equivalent to just a single epoch of LoRA fine-tuning across the corpus.

3. On-Demand Retrieval & Aggregation: Exploiting First-Order Additivity for Multi-Source Synthesis

Downstream reasoning tasks routinely require synthesizing evidence dispersed across multiple distinct textual sources. Because all stored knowledge atoms in the Gradient Bank are differentiated at the identical meta-learned coordinate \(\theta^*\), their linear sum strictly mirrors standard multi-instance gradient accumulation over the aggregate language-modeling loss. Upon receiving query \(q\), the retrieval module identifies the Top-\(K\) relevant documents and retrieves their corresponding gradient atoms \(\{g_{i_k}\}_{k=1}^K\). The system performs an instantaneous element-wise addition without invoking any expensive backward passes or online optimization loops:

\[g(q) = \sum_{k=1}^K g_{i_k}\]

4. Temporary Parameter Adaptation & Instant Rollback: Eliminating Weight Drift and Context Inflation

The aggregated gradient vector \(g(q)\) is scaled by the meta-learned per-tensor scalar step sizes \(\alpha^*\) to induce a query-specific temporary adapter state \(\theta_q = \theta^* - \alpha^* \odot g(q)\). The base model \(M(W, \theta_q)\) then generates the final prediction using only the query string \(q\) in its prompt. Because the input sequence is completely stripped of external document tokens, inference complexity remains strictly bounded to \(O(|q|^2h + |q|h^2)\), eliminating the quadratic memory and latency overheads that plague long-context RAG. Once generation completes, the system discards \(\theta_q\) and restores the unperturbed state \(\theta^*\). This architectural rollback prevents parameter shifts from accumulating across sequential queries, providing an impermeable barrier against catastrophic forgetting.

Loss & Training

During meta-training, training instances are structured as tuples \((\mathcal{C}, q, a)\), where \(\mathcal{C} = \{C_k\}_{k=1}^K\) denotes supporting context passages and \((q, a)\) represents the grounded question-answer pair. For each passage \(C \in \mathcal{C}\), the inner unsupervised loss optimizes the token-level cross-entropy:

\[\tilde{\mathcal{L}}_C(\theta) = -\frac{1}{T} \sum_{t=1}^T \log p_{(W, \theta)}(c_t \mid c_{<t})\]

The corresponding virtual update direction is computed as \(g_C = \nabla_\theta \tilde{\mathcal{L}}_C(\theta)\), and multi-passage gradients are summed into \(g_{\mathcal{C}} = \sum_{k=1}^K g_{C_k}\). The inner adaptation yields temporary parameters \(\theta_{\mathcal{C}} = \theta - \alpha \odot g_{\mathcal{C}}\), where \(\alpha\) contains learnable scalar step sizes initialized to \(10^{-2}\). The outer objective assesses conditional answer generation without exposing context passages \(\mathcal{C}\) in the prompt:

\[\mathcal{L}_{\mathrm{QA}}^{\mathcal{C}}(\theta_{\mathcal{C}}) = -\frac{1}{L} \sum_{t=1}^L \log p_{(W, \theta_{\mathcal{C}})}(a_t \mid q, a_{<t})\]

The complete bi-level optimization formulation is:

\[\min_{\theta, \alpha} \mathbb{E}_{(\mathcal{C}, q, a) \sim \mathcal{D}_{\text{meta}}} \left[ \mathcal{L}_{\mathrm{QA}}^{\mathcal{C}}\left( \theta - \alpha \odot \sum_{C \in \mathcal{C}} \nabla_\theta \tilde{\mathcal{L}}_C(\theta) \right) \right]\]

Meta-training is conducted for 1 epoch using bfloat16 mixed precision, a meta-batch size of 16, a maximum context length of 1,536 tokens, and a linear warmup scheduler reaching a peak outer learning rate of \(4 \times 10^{-4}\).

Key Experimental Results

Main Results

Evaluations span nine extensive benchmarks across general open-domain Wikipedia reasoning (2WQA, CWQ, HotpotQA) as well as legal reasoning (CaseHOLD, Learned Hands Family, HousingQA) and medical reasoning (PubMedQA, MedQA, BioASQ). Models are tested under the Top-1000 protocol on LLaMA-3.2-1B/3B and LLaMA-3.1-8B, comparing REGRAD (parameter-only adaptation) and REGRAD+ICL (hybrid parameter and context injection) against parametric CPT, RAG, and fine-tuned baselines.

Model Scale Method 2WQA (F1) CWQ (F1) HQA (F1) CaseHOLD (Acc) LHF (Acc) Housing (Acc) PubMed (Acc) MedQA (Acc) BioASQ (Acc) Overall Avg.
LLaMA-1B Direct 21.27* 31.71* 16.08* 42.1* 41.1* 64.3* 17.3* 6.24* 42.6* 31.41*
Standard CPT 15.34* 28.19* 11.42* 40.1* 40.3* 59.7* 18.9* 6.13* 46.6* 29.63*
Instruction CPT 30.71* 34.74* 18.93* 40.7* 47.1* 62.9* 20.1* 6.80* 41.4* 33.71*
RAG (ICL) 24.08* 33.58* 24.01* 42.7* 41.2* 65.8* 70.2* 7.88* 71.2* 42.29*
PE-RAG 24.98* 35.63* 27.94* 44.0* 43.2* 62.7* 62.9* 6.12* 66.3* 41.53*
Fine-tuned-PRAG 31.43* 45.26* 24.73* 65.1* 89.5* 81.9 92.4* 10.78 76.8* 57.54*
REGRAD (Ours) 34.61* 47.18* 30.72* 65.7 94.8 82.1 93.3 10.12 74.6* 59.24*
REGRAD + ICL 40.63 50.34 39.18 68.4 94.7 84.5 97.9 11.50 83.4 63.39
LLaMA-3B Direct 16.58* 36.69* 18.77* 55.8* 46.1* 47.7* 88.1* 10.63* 77.8* 44.24*
Standard CPT 10.96* 33.94* 15.48* 54.2* 44.5* 45.7* 78.4* 9.47* 75.1* 40.86*
Fine-tuned-RAG 35.48* 51.11 32.49* 50.9* 91.9 70.5* 86.4* 15.19* 82.1* 57.34*
Fine-tuned-PRAG 32.39* 51.42 29.51* 64.2* 89.2* 82.5 92.7 16.12 79.1* 59.68*
REGRAD (Ours) 36.82* 52.57 33.43* 68.8 94.7 83.1 92.8 16.66 78.6* 61.94*
REGRAD + ICL 44.18 54.08 43.47 71.3 94.6 84.8 97.9 18.21 85.7 66.03
LLaMA-8B Direct 33.42* 42.73* 26.34* 61.3* 54.1* 46.0* 91.2* 16.46* 81.2* 50.31*
Standard CPT 25.67* 40.24* 23.51* 58.1* 53.8* 46.2* 90.3* 15.59* 80.2* 48.18*
Fine-tuned-RAG 37.17* 54.21 39.16* 59.6* 91.9 69.5* 92.7* 23.22 84.6 61.34*
Fine-tuned-PRAG 35.71* 51.26* 32.97* 66.2* 78.6* 83.0 91.8* 21.37 82.6* 60.39*
REGRAD (Ours) 38.43* 56.41 36.31* 70.3* 96.2 84.8 93.2* 21.65 81.9* 64.36*
REGRAD + ICL 45.06 57.52 45.27 74.1 94.8 85.4 98.1 19.83* 85.3 67.26

Ablation Study & Empirical Analyses

1. Capability Preservation Under Continual Knowledge Injection (LLaMA-8B)

A shuffled 10,000-document corpus containing genuine supporting passages combined with random noise documents is incrementally injected into LLaMA-8B. The table tracks token-level F1 across sequential corpus expansion:

Document Scale 2Wiki (CPT) 2Wiki (RG) CWQ (CPT) CWQ (RG) HotpotQA (CPT) HotpotQA (RG) Benchmark Avg (CPT) Benchmark Avg (RG)
2.5K Docs 27.14 33.95 43.32 55.91 21.73 23.26 30.73 37.71
5.0K Docs 28.26 34.12 44.55 57.16 21.43 27.65 31.41 39.64
7.5K Docs 27.43 36.12 45.72 59.84 22.46 31.84 31.87 42.60
10.0K Docs 20.25 37.70 45.89 60.10 17.26 34.35 27.80 44.05

2. End-to-End Inference Latency Benchmarking (8×A100-80G Server)

Measured average execution time (in milliseconds) across 10 repeated runs over 100 randomly sampled general-domain test queries:

Serving Strategy LLaMA-1B (ms) LLaMA-3B (ms) LLaMA-8B (ms) Operational Profile
Direct Generation 13.44 18.36 28.30 Unaugmented forward pass, zero retrieved knowledge
Standard RAG 17.03 23.62 34.63 3 retrieved passages concatenated to prompt (token explosion)
REGRAD Cached 16.89 22.97 30.84 Fetches pre-computed atoms; tensor addition takes only 2.54 ms
REGRAD + ICL 18.21 26.60 38.10 Hybrid mode: combines low-rank update with prompt evidence
REGRAD Online 32.04 45.83 56.31 Zero-storage mode: backpropagates gradients on the fly

Key Findings

  • Mitigation of Cumulative Weight Drift: Under 10K incremental document injections, standard CPT experiences severe performance collapse (2Wiki F1 plunges from 28.26% down to 20.25%, with the 3-benchmark average declining from 31.41% to 27.80%). Conversely, REGRAD performance scales upward monotonically from 37.71% to 44.05%, validating that decoupling updates into retrievable units successfully halts catastrophic forgetting.
  • Criticality of the Meta-Learning Shaper: Injecting raw next-token prediction gradients without bi-level meta-training (Raw Gradient) consistently causes severe negative transfer—underperforming not only REGRAD but also zero-shot Direct Generation across all model scales, proving that unaligned language modeling gradients act as disruptive noise.
  • Semantic Specificity Over Task Triggering: Substituting retrieved gradients with randomly sampled atoms from the Gradient Bank yields only a marginal baseline improvement (activating general QA behavior), whereas injecting semantically aligned gradients produces substantial accuracy leaps, proving that knowledge atoms carry factual parametric data.
  • Superior Serving Efficiency Over In-Context RAG: Pre-computed REGRAD Cached incurs a negligible dynamic update merging overhead (merely 2.54 ms on LLaMA-8B), achieving faster end-to-end inference (30.84 ms) than context-expanded RAG (34.63 ms) while maintaining constant sequence length.

Highlights & Insights

  • Promoting Gradients to First-Class Retrievable Knowledge Entities: Transcends the traditional view of gradients as single-use optimization signals, successfully demonstrating that low-rank gradients can be cataloged, indexed, and retrieved as modular factual memory blocks.
  • Context-Stripped Outer Meta-Loss: Banning document context from the outer loss enforces an information-theoretic bottleneck, forcing the meta-optimization to route all factual knowledge through the adapter parameter shifts rather than in-context attention pathways.
  • Orthogonal Synergy with In-Context Retrieval: Demonstrates that parametric knowledge injection and non-parametric context injection are fundamentally complementary; their hybridization in REGRAD+ICL establishes new state-of-the-art benchmarks across 1B, 3B, and 8B model families.

Limitations & Future Work

  • Supervised Meta-Training Overhead: The initialization phase currently relies on downstream supervised QA pairs to align gradients, constraining its zero-shot utility in unannotated domains; exploring unsupervised or synthetic meta-objectives remains a critical next step.
  • Storage Overhead of the Gradient Bank: Storing per-document LoRA gradients scales linearly with corpus size (hundreds of kilobytes per document). While manageable via caching head documents and generating long-tail gradients on the fly, advanced gradient pruning, low-bit quantization, and sparse dictionary learning are needed.
  • Linear Gradient Merging Vulnerability: The unweighted additive combination of multiple gradients assumes first-order parameter orthogonality; under noisy retrieval conditions, distractor documents can induce destructive gradient interference.
  • vs Continual Post-Training (Gururangan et al., 2020a): CPT permanently alters shared network weights and inevitably degrades under sequential updates; REGRAD decouples knowledge ingestion into offline storage, applying reversible updates that preserve the foundational weights intact.
  • vs Standard RAG (Lewis et al., 2020): RAG operates purely via in-context evidence conditioning, suffering from attention dilution and quadratic prompt length growth; REGRAD injects knowledge directly into the weights, preserving base model sequence efficiency while remaining fully compatible with RAG prompts.
  • vs Parametric RAG (Su et al., 2025e): Parametric RAG requires training document parameters online; REGRAD employs bi-level meta-learning to enable instantaneous, one-step linear gradient accumulation without online backpropagation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering paradigm transforming gradients into retrievable, modular knowledge atoms with instant rollback]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluations across 3 model scales, 9 diverse benchmarks, detailed forgetting curves, and latency profiling]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear theoretical exposition with rigorous bi-level optimization formulation]
  • Value: ⭐⭐⭐⭐⭐ [Presents a groundbreaking, practical roadmap for lifelong LLM knowledge updating without catastrophic forgetting]