Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models¶

Conference: NeurIPS2026
arXiv: 2609.30935
Code: https://github.com/wangbing1416/EoupCT
Area: Optimization & Theory; continual fine-tuning of large language models
Keywords: continual learning, knowledge retention, differentiable pseudo data, soft prompts, gradient orthogonalization

TL;DR¶

EoupCT uses a frozen pretrained model and learnable soft prompts to construct differentiable knowledge proxies vulnerable to new-task updates, then combines distillation with gradient projection to protect historical tasks and general capabilities; experiments on SuperNI/MMLU subsets across six models improve retention, but its “first-order-only” and “absolute zero forgetting” claims require qualification.

Background & Motivation¶

Continual fine-tuning faces two distinct forms of forgetting: later tasks can reduce performance on earlier downstream tasks, and general knowledge acquired during pretraining can erode even when earlier tasks remain relatively stable. Historical task data or gradients can usually be recorded, whereas the original pretraining corpus and optimization trajectory of an off-the-shelf model are unavailable. Restricting new-task updates to the orthogonal complement of historical downstream gradients therefore does not automatically protect pretrained capabilities absent from that subspace.

Existing methods have different coverage gaps. OLoRA constrains the low-rank update subspaces of different tasks, CLoRA limits output perturbations through subspace regularization, and LoRAMoE and GainLoRA reduce interference with multiple branches and routing. These mechanisms can reduce conflicts during continual adaptation without identifying which general knowledge the current task will damage. Randomly generated replay text may also miss such vulnerabilities: readily generated knowledge may already be stable, while directions needing protection may rarely be sampled.

The paper uses the pretrained model itself as a source of knowledge proxies and adapts soft prompts to the current task, seeking representations that the student would struggle to preserve after a new-task update. “Estimating pretraining gradients” here means computing student gradients of a teacher-proxy distillation loss, not recovering recorded gradients from original training or obtaining original pretraining data. Core idea: identify vulnerable knowledge directions with task-dependent differentiable pseudo data, then incorporate them alongside historical task gradients into the constraints on continual fine-tuning updates.

Method¶

Overall Architecture¶

The inputs are a frozen pretrained teacher, a trainable student initialized from it, and a stream of downstream tasks; the experiments train only LoRA parameters. Each task receives a soft prompt that drives autoregressive generation of continuous surrogate embeddings by the teacher. Representation differences between teacher and student on the same surrogate sequence provide a retention signal, while the supervised new-task loss supplies the learning signal.

Training proceeds through “Differentiable Knowledge Proxy,” “Virtual-Step Vulnerability Search,” “Historical Null-Space Projection,” and “Conflict Correction & Joint Update.” The first two stages adjust the prompt and compute a protection gradient; the latter two adjust the student update direction. Freezing the teacher means its weights are not updated, not that the entire generation chain can be detached: prompt optimization still requires gradients through the teacher’s input computations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    T["Frozen pretrained teacher"] --> A["Differentiable<br/>Knowledge Proxy"]
    P["Learnable soft prompt"] --> A
    A --> B["Virtual-Step<br/>Vulnerability Search"]
    N["Supervised new-task gradient"] --> B
    B -.->|Optimize prompt only| P
    B -->|Protection gradient at actual student weights| C["Historical<br/>Null-Space Projection"]
    N --> C
    H["Historical downstream gradients"] --> C
    C --> D["Conflict Correction<br/>& Joint Update"]
    D --> S["Update student LoRA"]

The feedback edge represents training-time prompt optimization; only the bottom stage updates student parameters. Deploying the student does not require rerunning surrogate generation. Historical protection gradients support the prompt-diversity penalty, whereas historical downstream gradients support student null-space projection; these are separate memory banks.

Key Designs¶

1. Differentiable Knowledge Proxy: replace nondifferentiable text sampling with continuous surrogate embeddings

Direct generation of discrete tokens breaks the gradient path from the prompt to the generated sequence. The authors assign each task a soft prompt of length \(L\) and let the frozen teacher generate a sequence of length \(M\). At each step, vocabulary logits receive Gumbel noise and a temperature-controlled softmax produces weights; the next input is a weighted sum of token embeddings rather than a hard token.

\[ \pi_{m,i}=\frac{\exp((z_{m,i}+\varepsilon_{m,i})/\tau)}{\sum_j\exp((z_{m,j}+\varepsilon_{m,j})/\tau)},\qquad \mathbf e_m=\sum_i\pi_{m,i}(\mathbf W_{\mathrm{emb}})_i. \]

This lets the prompt influence the final protection loss through the entire autoregressive chain. Low temperature makes the weights closer to one-hot, but with Gumbel noise the limit corresponds to a noise-perturbed random categorical choice, not deterministic greedy decoding. Appendix A.1’s interpretation of the limit as a greedy token therefore requires caution and does not establish recovery of the true pretraining text distribution.

Teacher and student latent representations on these surrogate embeddings are matched with mean squared error; the protection gradient is the derivative of that loss with respect to the student’s trainable parameters. It measures the current student’s deviation from the teacher and is not the original pretraining objective, corpus gradient, or training history.

\[ \mathcal L_{\mathrm{pre}}(\boldsymbol\theta;\mathbf S(\mathbf P_t))=\ell_{\mathrm{MSE}}\big(\mathcal F_{\boldsymbol\theta^0}(\mathbf S(\mathbf P_t)),\mathcal F_{\boldsymbol\theta}(\mathbf S(\mathbf P_t))\big),\qquad \mathbf g_{\mathrm{pre}}(\mathbf P_t)=\nabla_{\boldsymbol\theta}\mathcal L_{\mathrm{pre}}. \]

Appendix D.5 actually applies the relaxation only to the 50 highest-probability candidate tokens at each step, rather than always using the full vocabulary. This reduces storage requirements but limits token coverage; changes in the candidate set also make the generation mapping not globally smooth. The proxy should therefore be understood as a local knowledge probe constrained by the prompt, sequence length, and candidate truncation.

2. Virtual-Step Vulnerability Search: target representation drift caused by the new task

Simply maximizing the current teacher–student discrepancy can find knowledge already forgotten without identifying what the new task is about to damage. The authors first compute the new-task gradient, temporarily move the student one step along it, and optimize the prompt against this virtual student. The actual student has not yet taken that update; after prompt search, the protection gradient is recomputed at the actual student weights.

\[ \boldsymbol\theta_{\mathrm{virt}}=\boldsymbol\theta-\alpha\mathbf g_{\mathrm{new}},\qquad \mathbf P_t^*=\arg\max_{\mathbf P_t}\left[\mathcal L_{\mathrm{pre}}(\boldsymbol\theta_{\mathrm{virt}};\mathbf S(\mathbf P_t))-\lambda\sum_{i<t}\cos^2\big(\mathbf g_{\mathrm{pre}}(\mathbf P_t),\mathbf g_{\mathrm{pre}}(\mathbf P_i)\big)\right]. \]

The second term penalizes squared cosine similarity between the current protection gradient and historical protection gradients so that successive tasks do not repeatedly probe the same knowledge. It encourages directional diversity, but a finite soft penalty neither guarantees strict orthogonality nor establishes semantic or distribution-wide coverage.

The authors use a first-order Taylor expansion to explain why the virtual step exposes conflict. The following expression makes prompt dependence explicit in the paper’s Eq. (8); it is a notational expansion, not an additional algorithm.

\[ \mathcal L_{\mathrm{pre}}(\boldsymbol\theta-\alpha\mathbf g_{\mathrm{new}};\mathbf S(\mathbf P_t))\approx\mathcal L_{\mathrm{pre}}(\boldsymbol\theta;\mathbf S(\mathbf P_t))-\alpha\langle\mathbf g_{\mathrm{new}},\mathbf g_{\mathrm{pre}}(\mathbf P_t)\rangle. \]

When virtual weights are treated as fixed, the virtual-loss term can indeed be optimized with ordinary prompt back-propagation without differentiating the new-task gradient. However, the baseline loss in the expansion also depends on the prompt. Maximizing virtual loss is therefore not unconditionally and exactly equivalent to maximizing gradient conflict; it may also prefer proxies with an already large distillation error.

More importantly, the full objective includes a cosine penalty on protection gradients. Since the protection gradient itself depends on the prompt, differentiating that term still involves mixed second derivatives between prompt and student parameters. Fixed virtual weights explain differentiation of the virtual-loss term alone, not a first-order-only full objective. The paper does not specify whether this term uses detachment, approximation, or a replacement objective, and the note does not assume the code has resolved the issue.

3. Historical Null-Space Projection: preserve recorded downstream task directions first

Both the current task gradient and the current protection gradient can affect earlier downstream tasks. The authors concatenate historical downstream gradients as columns and project both current gradients into their orthogonal complement. Projecting both rather than just the task gradient prevents the subsequently added distillation update from reintroducing historical-task conflict.

\[ \boldsymbol\Pi_{\mathrm{new}}=\mathbf I-\mathbf M_{\mathrm{new}}(\mathbf M_{\mathrm{new}}^\top\mathbf M_{\mathrm{new}})^{-1}\mathbf M_{\mathrm{new}}^\top,\qquad \tilde{\mathbf g}_{\mathrm{new}}=\boldsymbol\Pi_{\mathrm{new}}\mathbf g_{\mathrm{new}},\quad \tilde{\mathbf g}_{\mathrm{pre}}=\boldsymbol\Pi_{\mathrm{new}}\mathbf g_{\mathrm{pre}}^*. \]

The ordinary inverse in Eq. (9) requires linearly independent historical gradient columns. Empty historical memory should be interpreted as no historical constraint; repeated or degenerate directions can make the inverse nonexistent. Appropriate numerical treatment is needed, but the paper does not specify an implementation, so a pseudoinverse or regularization cannot be presented as an adopted author solution.

Algorithm 1 adds one gradient to each memory bank after each task, without clarifying whether it is a full-task average, the last batch, or a separate estimate. The protected objects are recorded directions, not all historical-example gradients at every later parameter state. As tasks accumulate, both memory storage and the excluded subspace grow, potentially reducing freedom to learn new tasks.

4. Conflict Correction & Joint Update: remove only the new-task component that harms the current proxy

After historical projection, the authors inspect the inner product of the task and protection gradients. A nonnegative value means new-task gradient descent does not increase proxy loss to first order; a negative value indicates conflict, requiring removal of the conflicting component along the protection gradient. The corrected task gradient is then added to the protection gradient.

\[ \tilde{\mathbf g}_{\mathrm{new}}^*=\begin{cases}\tilde{\mathbf g}_{\mathrm{new}}-\dfrac{\langle\tilde{\mathbf g}_{\mathrm{new}},\tilde{\mathbf g}_{\mathrm{pre}}\rangle}{\|\tilde{\mathbf g}_{\mathrm{pre}}\|^2}\tilde{\mathbf g}_{\mathrm{pre}},&\langle\tilde{\mathbf g}_{\mathrm{new}},\tilde{\mathbf g}_{\mathrm{pre}}\rangle<0,\\\tilde{\mathbf g}_{\mathrm{new}},&\text{otherwise},\end{cases}\qquad \boldsymbol\theta^+=\boldsymbol\theta-\eta(\tilde{\mathbf g}_{\mathrm{new}}^*+\tilde{\mathbf g}_{\mathrm{pre}}). \]

Appendix A.2 proves the nearest orthogonal gradient correction in Euclidean distance under conflict, not a global Pareto optimum of the entire nonconvex training problem. The joint update includes a component that reduces proxy distillation loss, so the final update need not be strictly orthogonal to the current protection gradient. The relevant requirements are a nonpositive first-order proxy-loss change and orthogonality to recorded historical directions.

These properties depend on current gradients, exact projection, and the corresponding actual update direction. They are local first-order guarantees, not guarantees of unchanged nonlinear outputs or absolute zero forgetting after finite updates. Appendix D.5 additionally uses AdamW: coordinate-wise adaptive scaling, momentum, or weight decay applied after projection can move the actual parameter increment out of the same null space. The paper does not explain how the optimizer preserves the projection guarantee.

Loss & Training¶

The new task uses autoregressive negative log-likelihood of target answers, while the student also carries the teacher-representation distillation objective. The prompt performs maximization and the student performs minimization; this is not one loss that updates all parameters through a single backward pass. Each iteration follows new-task gradient computation, virtual prompt search, actual protection-gradient computation, two-stage projection, and student update; memory is expanded after the task ends.

If teacher and student are identical, the mean squared error and its student gradient can be zero; projection can also reduce a nonzero protection gradient to zero. Squared-cosine normalization may then be undefined, and the conflict-projection denominator must be interpreted under a nonzero condition. The paper does not fully specify initialization or numerical safeguards for these degenerate cases, so implementation details cannot be inferred from the formulas.

The experiments use LoRA rank 4, AdamW learning rate \(2\times10^{-4}\), gradient checkpointing, a training input limit of 1024 tokens, and an answer-generation limit of 50 tokens. MMLU uses up to 5-shot contexts with a 2048-token limit; demonstrations are reduced when necessary, batch size is 1, and final-position logits select among candidate answers.

The sensitivity ranges for prompt and surrogate lengths are \(\{4,8,16,32\}\), and the appendix gives surrogate lengths of 16 or 32 as examples. The paper does not fully disclose prompt-search inner loops, temperature schedules, virtual step sizes, or diversity coefficients for every model; sensitivity curves cannot be treated as a universal default configuration.

Key Experimental Results¶

Main Results¶

The continual-learning stream contains 15 SuperNI tasks spanning question answering, information extraction, sentiment analysis, summarization, and dialogue, with three randomized task orders. General capabilities are evaluated on 9 MMLU subjects grouped into STEM, Humanity, and Other, not a demonstrated evaluation of the full MMLU benchmark. Table 1 averages MMLU results over the three task orders but reports SuperNI separately by order.

SuperNI uses ROUGE-L for generation quality. Its Fgt. averages each task’s score immediately after learning it minus its score after the entire stream; MMLU Fgt. is original-model accuracy minus final accuracy. Both are score differences, not relative percentages; negative forgetting means the final score exceeds the corresponding baseline.

\[ \mathrm{FR}_{\mathrm{SuperNI}}=\frac1T\sum_{t=1}^{T}(R_{t,t}-R_{T,t}),\qquad \mathrm{FR}_{\mathrm{MMLU}}=A_0-A_T. \]

The following excerpts the Order 1 and STEM columns of the paper’s Table 1, retaining representative comparisons across models; it is not an aggregate over every task order and MMLU subject.

Model	Method	SuperNI ROUGE-L ↑	SuperNI Fgt. ↓	MMLU STEM Acc. ↑	MMLU STEM Fgt. ↓
Qwen3-4B	LoRA	45.1	13.7	56.4	4.5
Qwen3-4B	OLoRA	48.9	9.0	58.1	2.9
Qwen3-4B	EoupCT	50.7	6.0	60.2	0.7
Llama3-8B	OLoRA	44.6	12.8	43.6	2.4
Llama3-8B	EoupCT	51.4	6.5	45.0	1.0
Gemma2-9B	OLoRA	50.0	8.8	42.9	10.3
Gemma2-9B	EoupCT	53.4	4.8	52.3	0.9

For example, Qwen3-4B gains 1.8 task-score points over OLoRA and reduces task forgetting by 3.0 points; Gemma2-9B gains 9.4 percentage points in STEM accuracy. These results support improved retention under this protocol, not accurate recovery of unknown pretraining gradients.

Ablation Study¶

The following gives the Qwen3-4B columns of Table 2. C2 denotes vulnerability search, P&PO protection-gradient diversity, N&PO the new-task/protection-gradient conflict constraint, and N&NO historical downstream-gradient constraints. Table 2 abbreviates its MMLU columns as Acc./Fgt.; full-model values match Table 1’s STEM columns. The original labels are therefore retained rather than interpreting them as averages across all 9 subjects.

Config	SuperNI ROUGE-L ↑	SuperNI Fgt. ↓	MMLU Acc. ↑	MMLU Fgt. ↓
EoupCT	50.7	6.0	60.2	0.7
w/o C2	47.6	10.3	59.0	2.0
w/o P&PO	47.3	9.2	57.8	3.1
w/o N&PO	47.4	8.7	57.9	3.0
w/o N&NO	46.9	10.3	56.9	4.1

Removing vulnerability search reduces the task score by 3.1 points; removing historical downstream protection reduces it by 3.8 points and increases MMLU forgetting from 0.7 to 4.1. This supports both targeted proxy search and historical-task protection, but ablations alone cannot establish strict satisfaction of the full theoretical constraints.

Appendix Table 3 reports runtime in minutes under the same 8-GPU setting. Minutes are retained instead of recalculating the authors’ rounded multipliers, avoiding confusion among relative costs in different columns.

Model	LoRA	OLoRA	CLoRA	EoupCT
Qwen3-4B	62	204	155	129
Llama3-3B	35	151	114	65
Gemma2-9B	76	271	210	178
Author-reported average	60	203	157	122

EoupCT is faster than OLoRA and CLoRA in the reported configurations but clearly slower than ordinary LoRA: an average of 122 minutes versus 60 minutes cannot be summarized as negligible training overhead. The table insufficiently specifies GPU models, peak memory, and matched-budget details; costs cannot be directly extrapolated to a single GPU or longer proxies.

Key Findings¶

Results across three task orders and three model families support robustness, but do not replace error bars or significance tests from repeated independent training with a fixed order.
Very short prompts and proxies perform worse; increasing length helps, while reaching 32 may slightly reduce performance. The cache provides curve descriptions, so unavailable exact curve values are not invented.
Table 1 reports Llama3-3B STEM Fgt. of -0.8, indicating possible positive transfer on that subset, not improvement of all general knowledge.
“Removing any component worsens every forgetting metric” does not hold cell by cell: Table 2 gives Gemma2-2B SuperNI Fgt. of 10.1 without C2 versus 10.3 for the full model. Overall task-score deterioration must be distinguished from monotonic worsening of every forgetting value.

Highlights & Insights¶

The task determines what is protected. Pseudo data serves not merely as inexpensive replay but as a probe of teacher–student discrepancies vulnerable to the current update, making retention task-specific.
The two historical memories serve different purposes. Historical protection gradients diversify knowledge-probe directions; historical downstream gradients directly constrain student updates. Separating them clarifies that proxy diversity and old-task stability are different objectives.
Projection before conflict correction has a purpose. Both gradients enter the same historical null space before their mutual conflict is corrected, preventing local knowledge protection from undoing existing downstream constraints.

Limitations & Future Work¶

A knowledge proxy is not the pretraining distribution. Short prompts, short sequences, and top-50 truncation limit coverage; proxies may reside in soft-embedding rather than natural-text space, and teacher generation preferences can omit knowledge that is difficult to generate. Natural-text replay versus soft-embedding protection and broader evaluation deserve study.
The theory exceeds the demonstrated evidence. Appendix A.3’s global bound requires prompt coverage of the true distribution, a worst-case prompt, and control of the worst-case loss at the required weights. A worst-case prompt at virtual weights need not remain worst-case at updated weights; Remark 3.1’s local-representativeness explanation does not complete the global coverage proof.
First-order efficiency has unresolved conditions. The prompt derivative of the full gradient-cosine penalty involves mixed second-order terms. Appendix B.1’s blanket description of Hessian-vector products as requiring quadratic memory is not generally valid: automatic differentiation need not materialize the Hessian. Appendix B.2 also calls quadratic complexity “exponential,” which is incorrect terminology.
Degeneracy and actual updates need clarification. Zero protection gradients, a singular historical Gram matrix, actual AdamW increments, and stale historical directions affect whether projection constraints hold. Future work should disclose numerical treatment and measure actual-update constraint residuals rather than rely solely on formal orthogonality.
Experimental conclusions remain bounded. Evidence covers a short task stream, limited model scales, and an MMLU subset, without adequate significance statistics. The text claims consistent deterioration under component removal despite a Table 2 counterexample; Appendix D.2 uses both subset and “full MMLU” wording, requiring clarification of scope.
Reproduction details are incomplete. The author label “Llama3-3B” lacks an exact checkpoint and must not be silently replaced with another model version. Prompt-search details and memory configurations are also insufficiently specified. Appendix E’s dynamics matrices and caption summaries should not be assumed to share an aggregation rule without reconciliation.

vs OLoRA / CLoRA: These primarily constrain perturbations through low-rank weights or subspaces; EoupCT additionally constructs task-dependent teacher proxies and adjusts gradient directions, at the cost of generation, prompt search, and protection-gradient computation.
vs LoRAMoE / GainLoRA: Multiple experts and gates reduce conflict through parameter specialization; EoupCT emphasizes knowledge constraints during a single student’s updates. Runtime advantages apply only to Table 3’s configurations, not all routing approaches.
vs SSR / LAMOL: Pseudo-rehearsal typically synthesizes historical-task examples for joint training; this method generates differentiable embedding proxies and searches vulnerable directions using virtual updates. The transferable idea is a task-targeted retention probe, not a claim of recovering original training data.

Rating¶

Novelty: 4/5 — Connects differentiable knowledge proxies, virtual-step search, and continual gradient projection around a clearly identified problem.
Experimental Thoroughness: 3/5 — Multiple models, task orders, and ablations, but limited general-capability coverage, statistics, and reproduction details.
Writing Quality: 2/5 — The workflow is understandable, but first-order-only, global-protection, and absolute-zero-forgetting claims are overstated.
Value: 4/5 — Offers a research direction for retention without original pretraining data, with implementation and theoretical calibration still needed.