Skip to content

CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/kwang-min-ki/CPR
Area: LLM Reasoning
Keywords: Catastrophic Forgetting, Domain Adaptation, Token-level Routing, Critical-Point Dispatch, Collaborative Decoding

TL;DR

Addressing the severe catastrophic forgetting of general reasoning capabilities during supervised fine-tuning (SFT), CPR (Critical-Point Routing) decouples the general base model from the domain expert at the model level, employing a lightweight hierarchical router that dynamically invokes the expert only on critical tokens where the base model fails, achieving a 1.4%–5.5% domain gain over vanilla SFT while suppressing general-capability degradation from up to 14.5% down to at most 0.5% with ~30% sparse expert invocation.

Background & Motivation

Large language models (LLMs) pre-trained on diverse web-scale corpora demonstrate impressive natural language understanding and multi-step reasoning capabilities. When adapting LLMs to specialized vertical domains such as mathematical problem solving or biomedical clinical question answering, supervised fine-tuning (SFT) has established itself as the de facto operational standard. However, an extensive body of empirical research reveals that domain-targeted SFT inevitably triggers severe catastrophic forgetting, causing model performance on general capabilities—such as multidisciplinary knowledge benchmarks (MMLU), commonsense reasoning (CommonsenseQA), and challenge-level scientific reasoning (ARC-C)—to deteriorate drastically. This degradation is fundamentally detrimental because vertical domain tasks themselves do not operate in a vacuum; they heavily lean upon foundational general competencies such as instruction following, context parsing, and coherent multi-step deduction. Weakening these foundation abilities causes the fine-tuned specialist to become brittle and failure-prone in real-world deployments.

To combat this vulnerability, existing paradigms predominantly pursue single-model loss regularizations that penalize deviations from the base model during fine-tuning. For example, DFT rescales cross-entropy loss by predicted token probabilities to neutralize an implicit inverse-probability reward bias, EAFT incorporates entropy-guided soft gating to suppress destructive gradient updates on confident-conflict tokens, and LfU regularizes internal representations against simulated undesirable update directions. While these techniques slow the rate of forgetting, they share an intrinsic structural limitation: both domain-specialized and general knowledge are forcibly compressed into a single shared set of parameters. Empirical analyses consistently show that all single-model regularization techniques remain bounded along a rigid negative-slope Pareto trade-off curve, merely choosing different compromise operating points without escaping the single-weight frontier.

This paper challenges the fundamental premise of single-weight parameter compression and re-frames the problem as an architectural decoupling challenge. Recognizing that the pre-trained base model already preserves robust general reasoning abilities, the authors propose to leave the base model intact and frozen, selectively consulting the SFT expert exclusively when deep domain expertise is genuinely indispensable. This insight establishes an adaptive, token-level multi-model collaborative decoding framework. Core idea: Decouple domain-specific expertise from general foundational capabilities at the model level by designating tokens where the base model fails but the domain expert succeeds as "critical points," training a lightweight hierarchical router to predict expert necessity, and executing sparse collaborative decoding stabilized by momentum smoothing and threshold-gated 3-way dispatch.

Method

Overall Architecture

CPR addresses the challenge of preserving general competencies during domain adaptation without incurring the prohibitive computational and memory overhead of constant dual-model decoding (such as naive Ensembling or Contrastive Decoding). Given an input sequence, at each autoregressive generation step \(t\), the frozen base model \(f_B\) always executes a forward pass to produce the last-layer hidden state \(h_t^B\). A lightweight hierarchical router processes both sequence-level macro context and step-level micro features to estimate the raw probability of expert necessity \(p_t\). This raw probability is temporally smoothed via exponential moving average to yield a stabilized momentum metric \(m_t\). A threshold-gated 3-way dispatch mechanism then maps \(m_t\) into an execution weight \(\lambda_t\), dynamically selecting between pure base decoding, soft probability mixture decoding, and pure expert decoding. Whenever the expert is awakened after multiple inactive steps, a single batched KV cache catch-up pass synchronizes historical activations, minimizing wall-clock decoding latency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input prompt and generated token sequence x_<t"] --> B["Base model fB forward pass<br/>Extract last-layer hidden state h_t^B"]
    B --> C
    subgraph ROUTER["Hierarchical Router g_phi: Dual-granularity Criticality Estimation"]
        direction TB
        C["Hierarchical Router g_phi<br/>Macro Encoder MLP_macro extracts question summary z_dom<br/>Micro Router MLP_micro combines h_t^B and z_dom to output p_t"]
    end
    C --> D["Momentum Smoothing Filter<br/>Exponential moving average m_t = alpha*m_{t-1} + (1-alpha)*p_t<br/>Absorbs short-range noise and suppresses boundary oscillations"]
    D --> E
    subgraph DISPATCH["Threshold-gated 3-way Dispatch Mechanism"]
        direction TB
        E{"Threshold-gated dispatch on m_t"}
        E -->|m_t < tau_low| F["Hard Base Regime<br/>lambda_t = 0: Pure base output, expert remains dormant"]
        E -->|tau_low <= m_t <= tau_high| G["Soft Blend Regime<br/>lambda_t = m_t: Expert invoked, convex mixture of distributions"]
        E -->|m_t > tau_high| H["Hard Expert Regime<br/>lambda_t = 1: Expert invoked, pure expert output distribution"]
    end
    F --> I["Output distribution sampling p(x_t|x_<t)<br/>Re-awakened expert synchronizes skipped steps via single batched KV catch-up"]
    G --> I
    H --> I

Key Designs

1. Critical-point Labeling & Hierarchical Router Architecture: Framing expert invocation as fine-grained critical classification

Query-level routers cannot satisfy the nuanced requirements of complex reasoning, where foundational transitions and explanatory clauses are best handled by the generalist base model while arithmetic computations and domain formulations require the specialist. CPR formalizes "critical points" via automated teacher-forcing comparison on domain SFT data between base greedy prediction \(\hat{x}_t^{(B)} = \arg\max_v p_b(v|x_{<t})\) and expert greedy prediction \(\hat{x}_t^{(E)} = \arg\max_v p_e(v|x_{<t})\). A token position is designated as a critical point (\(z_t = 1\)) strictly when the base model errs while the expert predicts the true ground-truth token \(x_t\). If the base model already predicts correctly, \(z_t = 0\) regardless of expert accuracy (as calling the expert is superfluous). Positions where both fail are marked as invalid supervision \(\emptyset\). To accurately predict \(p_t\), a two-tier hierarchical router \(g_\phi\) is engineered: - Macro Encoder: Operates once per sequence. It mean-pools the base model's last-layer hidden states across the question span \(S_q\), yielding query context \(c = \frac{1}{|S_q|} \sum_{i \in S_q} h_i^B\), which a 2-layer ReLU MLP projects into domain summary vector \(z_{\text{dom}} = \text{MLP}_{\text{macro}}(c) \in \mathbb{R}^{d_h}\). This global prior prevents spurious expert activations on out-of-domain queries. - Micro Router: At each generation step \(t\), the current base hidden state \(h_t^B \in \mathbb{R}^d\) is concatenated with \(z_{\text{dom}}\), passing through a 2-layer ReLU MLP with Sigmoid activation to predict scalar invocation probability:

\[p_t = g_\phi(h_t^B, c) = \sigma\left(\text{MLP}_{\text{micro}}\left([h_t^B; z_{\text{dom}}]\right)\right) \in (0, 1)\]

Given that critical points are sparse across full response sequences (\(z_t = 0\) tokens heavily outnumber \(z_t = 1\)), the router is optimized using re-weighted binary cross-entropy with a square-root positive class weight:

\[\mathcal{L}(\phi) = -\frac{1}{|V|} \sum_{t \in V} \left[ w_+ z_t \log p_t + (1 - z_t) \log(1 - p_t) \right], \quad w_+ = \sqrt{\frac{N_0}{N_1}}\]

This square-root formulation balances the gradient signal while moderating the destabilizing over-weighting caused by standard inverse-frequency weighting.

2. Momentum Smoothing & Threshold-gated 3-way Dispatch: Eliminating high-frequency jitter and marginalizing boundary ambiguity

Relying on raw instantaneous probabilities \(p_t\) with a rigid 0.5 binary threshold induces severe decoding collapse: minor fluctuations in hidden representations cause erratic switching between base and expert across adjacent tokens, disrupting textual coherence. Furthermore, binary hard gating ignores the mutual distributional benefits of model ensembling in borderline contexts. CPR resolves this with a dual-stage stabilization pipeline: First, an exponential moving average (EMA) filter computes running momentum: \(m_0 = 0, m_t = \alpha m_{t-1} + (1 - \alpha) p_t\), using default momentum decay \(\alpha = 0.5\). This acts as a low-pass filter, dampening short-range volatility while preserving rapid switching responsiveness when encountering genuine critical segments. Second, two decision thresholds \(\tau_{\text{low}} = 0.35\) and \(\tau_{\text{high}} = 0.65\) segment \(m_t\) into three operational regimes:

\[\lambda_t = \begin{cases} 0, & \text{if } m_t < \tau_{\text{low}} \quad (\text{Hard Base regime}) \\ m_t, & \text{if } \tau_{\text{low}} \le m_t \le \tau_{\text{high}} \quad (\text{Soft Blend regime}) \\ 1, & \text{if } m_t > \tau_{\text{high}} \quad (\text{Hard Expert regime}) \end{cases}\]

In the ambiguous intermediate regime (\(\tau_{\text{low}} \le m_t \le \tau_{\text{high}}\)), the dispatch weight smoothly equals \(m_t\) itself, constructing a convex combination \(p(x_t|x_{<t}) = (1 - \lambda_t) p_b(x_t|x_{<t}) + \lambda_t p_e(x_t|x_{<t})\). From a latent-variable perspective, treating router output as the posterior probability of expert selection makes this convex mixture the exact result of marginalizing over the latent model choice.

3. Selective Expert Invocation & Batched KV Cache Catch-up: Circumventing memory-bandwidth decoding bottlenecks

Standard collaborative decoding frameworks (such as Ensembles and Contrastive Decoding) evaluate both models at every single autoregressive step, inevitably doubling forward-pass computation and memory access. In CPR, although the base model always executes to feed the router, the expert model remains dormant during Hard Base regimes, participating in generation only during Soft Blend and Hard Expert states (amounting to ~30% of total tokens). When the expert is awakened after an arbitrary sequence of inactive steps (e.g., 7 consecutive skipped positions), its autoregressive key-value cache lacks the historical context necessary for causal self-attention. Instead of stepping through sequential forward passes, CPR executes a single batched KV cache catch-up forward pass: all skipped token representations are passed concurrently through the expert layers to populate the KV cache in parallel, directly proceeding to the current single-step generation. Because autoregressive decoding with small batch sizes is predominantly memory-bandwidth-bound rather than compute-bound, amortizing memory load overhead via a single parallel catch-up pass dramatically reduces wall-clock latency compared to always-on dual decoding.

Loss & Training

  • Backbone & Expert Construction: Evaluated on Gemma3-4B and Llama3.1-8B backbones. Mathematical experts are trained on the GSM8K 8K split (0-shot CoT). Medical experts are trained on an 8K balanced subset of PubMedQA (4K yes / 4K no, 2-shot CoT with randomly shuffled candidate options to eliminate positional bias; demonstrations sampled from MMLU). Hyperparameters: sequence length 1024, effective batch size 16, learning rate 2e-5, constant schedule, 6 training epochs.
  • Router Training Setup: Base and expert models are kept completely frozen; only the hierarchical router parameters \(\phi\) (\(d_h = 256\), 2-layer MLPs) are trained. Optimizer: sequence length 1024, batch size 16, learning rate 1e-4, warmup ratio 0.1, 6 epochs. Macro pooling is strictly restricted to the question span, preventing shared few-shot demonstrations from diluting question-specific domain signals.

Key Experimental Results

Main Results

The primary evaluation spans mathematical reasoning (GSM8K, ASDiv, SVAMP), medical QA (PubMedQA, MedQA, CareQA), and three locked general capability benchmarks (MMLU, CommonsenseQA, ARC-C) under zero-shot and few-shot CoT prompting.

Math Domain & General Domain Evaluation (Gemma3-4B & Llama3.1-8B)

Method GSM8K (In-Domain) ASDiv (OOD Math) SVAMP (Robust Math) Math Average MMLU CSQA ARC-C General Average Overall Average
Gemma3-4B
Base 29.11 44.51 48.67 40.76 54.16 59.13 64.42 59.24 50.00
SFT (Expert) 51.25 58.00 50.67 53.31 (+12.55) 49.17 52.91 55.38 52.49 (-6.75) 52.90 (+2.90)
DFT (Reg.) 32.84 48.55 42.33 41.24 (+0.48) 45.95 57.66 47.59 50.40 (-8.84) 45.82 (-4.18)
EAFT (Reg.) 50.04 61.92 57.67 56.54 (+15.78) 51.15 48.87 54.10 51.37 (-7.87) 53.95 (+3.95)
LfU (Reg.) 37.39 53.67 43.33 44.80 (+4.04) 50.41 53.56 53.94 52.64 (-6.60) 48.72 (-1.28)
Switch Gen. (Patch) 46.93 57.43 49.67 51.34 (+10.58) 50.90 52.90 57.25 53.68 (-5.56) 52.51 (+2.51)
RouteLLM (Query) 49.63 57.87 50.67 52.72 (+11.96) 49.91 52.99 56.01 52.97 (-6.27) 52.85 (+2.85)
Ensemble (Dual) 53.08 60.38 54.00 55.82 (+15.06) 52.98 59.95 61.95 58.29 (-0.95) 57.06 (+7.06)
Contrastive Dec. 45.41 50.28 52.00 49.23 (+8.47) 54.63 59.71 65.87 60.07 (+0.83) 54.65 (+4.65)
CPR (Ours) 49.58 62.60 56.00 56.06 (+15.30) 55.89 61.10 67.24 61.41 (+2.17) 58.74 (+8.74)
Llama3.1-8B
Base 34.72 35.84 44.33 38.30 57.23 61.18 71.67 63.36 50.83
SFT (Expert) 45.87 58.79 60.00 54.89 (+16.59) 44.57 52.74 49.40 48.90 (-14.46) 51.90 (+1.07)
DFT (Reg.) 41.67 57.91 59.87 53.15 (+14.85) 47.90 60.52 46.28 51.57 (-11.79) 52.36 (+1.53)
EAFT (Reg.) 47.23 62.43 63.00 57.55 (+19.25) 43.20 53.07 45.17 47.15 (-16.21) 52.35 (+1.52)
LfU (Reg.) 56.56 61.87 58.67 59.03 (+20.73) 56.36 65.50 67.49 63.12 (-0.24) 61.08 (+10.25)
Switch Gen. (Patch) 46.10 58.27 60.33 54.90 (+16.60) 47.47 52.98 51.01 50.49 (-12.87) 52.69 (+1.86)
RouteLLM (Query) 45.72 59.56 61.00 55.43 (+17.13) 45.43 53.33 49.23 49.33 (-14.03) 52.38 (+1.55)
Ensemble (Dual) 51.30 62.80 64.00 59.37 (+21.07) 52.39 60.14 56.71 56.41 (-6.95) 57.89 (+7.06)
Contrastive Dec. 49.60 60.89 61.33 57.27 (+18.97) 51.61 59.28 60.05 56.98 (-6.38) 57.13 (+6.30)
CPR (Ours) 51.80 62.70 66.67 60.39 (+22.09) 58.97 64.26 65.25 62.83 (-0.53) 61.61 (+10.78)

Ablation Study

Ablation analysis on router components, momentum decay, and dispatch thresholds (Averaged across Gemma3-4B and Llama3.1-8B)

Configuration Math Average (%) General Average (%) Overall Average (%) Analysis & Insights
Base Model Baseline 39.53 61.30 50.42 Unadapted generalist base performance
Router Architecture
w/o Macro Encoder 57.52 60.67 59.10 General average drops by 1.5%, showing macro query embedding acts as a necessary domain prior
w/o Micro Router 49.63 61.28 55.45 Math domain drops sharply by 8.6%, proving coarse query routing fails to pinpoint critical tokens
Hierarchical Router (Default) 58.23 62.12 60.18 Synergistic combination achieves optimal balance across both axes
Momentum Decay \(\alpha\)
\(\alpha = 0.0\) (No Smoothing) 55.97 60.55 58.26 Decision boundaries oscillate with high-frequency noise, harming both domains
\(\alpha = 0.5\) (Default EMA) 58.23 62.12 60.18 Optimal balance between historical inertia and prompt switching sensitivity
\(\alpha = 0.9\) (Excessive Inertia) 46.32 62.71 54.52 Heavy inertia causes delayed expert entry at critical transitions (-11.9% math drop)
Dispatch Thresholds \((\tau_{\text{low}}, \tau_{\text{high}})\)
\((0.5, 0.5)\) (Hard Switching) 51.98 62.35 57.16 Eliminating the soft-blend buffer degrades domain reasoning (-6.3% on math)
\((0.35, 0.65)\) (Default 3-way) 58.23 62.12 60.18 Balances probabilistic marginalization with sparse expert execution
\((0.0, 1.0)\) (Fully Soft Blend) 58.07 62.26 60.17 Comparable accuracy but requires running the expert on 100% of tokens, losing speedups

Key Findings

  • Escaping the Single-Weight Pareto Frontier: Linear regression analysis across single-model regularization techniques yields consistent negative trade-off slopes ranging from -0.14 to -0.44. In contrast, CPR breaks outside this trade-off: on Llama3.1-8B math, vanilla SFT drops general capability by 14.5%, whereas CPR limits this degradation to just 0.5% while outperforming the SFT expert in domain accuracy by an additional 5.5%.
  • Token-level Sparsity and Latency Profiling: Expert invocation is highly concentrated on domain-dense tokens (arithmetic operators, numerals, clinical terms), averaging only ~30% of total tokens. On Gemma3-4B GSM8K decoding, full dual-model Ensemble incurs 1.89× latency (9.4 tokens/s), while CPR achieves 1.40× latency (13.4 tokens/s) including batched KV catch-up passes.
  • Zero-Shot Generalization to External Experts: When applied post-hoc to independently trained public experts (OpenMath2-Llama3.1-8B and Qwen2.5-Math-1.5B), CPR successfully learns routers without modifying expert weights. Notably, for Qwen2.5-Math-1.5B—where extreme SFT forgetting drops general accuracy by 23.33% (yielding a net overall loss of -7.79%)—CPR mitigates the general drop to -0.58%, turning the deployment into a positive net gain (+1.78%).

Highlights & Insights

  • Architectural Decoupling as a Principled Fix for Catastrophic Forgetting: Instead of struggling to find regularized loss functions that compress competing distributions into a single weight vector, preserving the pristine base model and treating the SFT expert as an on-demand accelerator completely removes parameter interference.
  • Momentum Smoothing plus Convex Marginalization: Autoregressive decoding is highly sensitive to token-to-token variance. Combining low-pass EMA filtering with a 3-way threshold gate provides a Bayesian interpretation: it acts as a hard switch in high-confidence regimes and performs posterior marginalization in ambiguous transition zones.
  • LoRA-Adapter Hybrid for Low Memory Footprint: In resource-constrained environments, parameterizing the expert as a LoRA adapter over the shared frozen base reduces CPR GPU memory overhead from 16.60 GB (dual full models) down to 8.83 GB (close to the single-model 8.07 GB baseline) with only minor latency overhead (1.49×).

Limitations & Future Work

  • Permanent Base Model Compute Overhead: Because the router takes the base model's last-layer hidden states \(h_t^B\) as input, the base model must execute on every step, establishing a strict lower bound on per-step inference latency. Exploring prefix-only routing or ultra-compact external token probes could decouple the router from base model forward passes.
  • Assumption of Expert Asymmetric Advantage: Critical-point labeling relies on identifying tokens where the expert succeeds while the base model fails. If the domain expert is sub-optimally trained and fails to demonstrate a decisive performance margin on a meaningful token subset, router supervision degenerates.
  • Extension to Architecturally Heterogeneous Models: The current micro router and batched KV cache catch-up pass presume identical tokenizer vocabularies and shared model structural dimensions. Adapting CPR to heterogeneous pairings (e.g., a compact local specialist routing into a massive generalist model) remains an open challenge.
  • vs DFT / EAFT / LfU (Loss Regularization Baselines): Conventional methods modify loss gradients to curb parameter drift during SFT, inevitably getting trapped along the single-weight domain-general trade-off curve; CPR decouples both capabilities at the model level, breaking free of parameter-sharing bottlenecks.
  • vs RouteLLM / FrugalGPT (Query-level Routing): Query-level dispatch makes a coarse binary decision for the entire sequence. Once a query is routed to the domain expert, all generated tokens are subject to catastrophic forgetting; CPR operates at token-level granularity, keeping connective reasoning anchored in the base model.
  • vs Switch Generation / Co-LLM (Patch-level & Collaborative Decoding): Patch-level switching operates on rigid block segments, whereas Ensemble and Contrastive Decoding incur 2× compute and memory bandwidth costs by evaluating both models at every step. CPR's 3-way dispatch with sparse KV catch-up achieves superior accuracy with substantially lower latency overhead.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ (Reframes catastrophic forgetting from an intractable loss-level trade-off into an elegant architectural decoupling and token-level routing framework)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive validation across Gemma3/Llama3/Qwen2.5 backbones, math/medical/finance/instruction domains, complete with ablation and efficiency profiling)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, coherent narrative, comprehensive empirical visualization, and rigorous causal analysis)
  • Value: ⭐⭐⭐⭐⭐ (Provides an immediately actionable, post-hoc plug-and-play paradigm for deploying highly specialized yet uncorrupted enterprise reasoning LLMs)