KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/meghanadhpulivarthi/KItCAT
Area: LLM Reasoning
Keywords: Knowledge Injection, Continued Pre-training, Auto-regressive Training, Input Corruption, Regularization
TL;DR¶
KItCAT presents a lightweight and compute-free data augmentation framework for decoder-only continued pre-training (CPT) by stochastically corrupting input token prefixes while preserving the original autoregressive targets, eliminating spurious lexical shortcuts in low-diversity niche corpora and achieving a 5–9× effective data multiplier over standard next-token prediction.
Background & Motivation¶
Large Language Models (LLMs) assimilate colossal amounts of generalized world knowledge across extensive pre-training corpora, yet routinely falter when tasked with answering domain-specific inquiries drawn from niche, proprietary, or private materials like enterprise technical manuals and internal engineering documentations. Continued Pre-Training (CPT) is widely adopted as the standard paradigm to inject such specialized factual knowledge directly into model parameters. However, specialized niche documents rarely repeat facts or present them in varied syntactic contexts. In such low-diversity regimes, training autoregressive decoder-only LLMs over multiple epochs exposes the network to identical input prefixes and completion targets repeatedly, causing the model to latch onto superficial lexical correlations and spurious co-occurrence patterns rather than internalizing the underlying semantic knowledge.
To inject expressive diversity into low-resource domain corpora, contemporary state-of-the-art approaches typically employ powerful proprietary LLMs (e.g., GPT-4) to synthetically generate extensive sets of paraphrases or extract structured entity-relation triples (such as Entigraph). While demonstrably effective at diversifying lexical contexts, this reliance on external synthetic generation introduces crippling computational bottlenecks: generating 5× to 10× paraphrased corpora demands massive inference budgets and API expenditures, while also introducing severe compliance hurdles in privacy-sensitive enterprise environments (such as healthcare, defense, and finance). This creates a critical architectural tension: can we induce potent data diversity natively during auto-regressive training without depending on external generative models?
The authors ground their inquiry on a foundational insight: overfitting in low-diversity CPT regimes is fundamentally exacerbated by the deterministic recurrence of identical sequences across training epochs. When a model grasps a spurious surface pattern during early epochs, repeated identical exposures relentlessly reinforce that shortcut, degrading out-of-distribution question-answering generalization. Core idea: prevent the decoder-only model from ever seeing the exact same input prefix twice by stochastically corrupting conditioning token sequences while keeping the original next-token prediction targets unchanged, thereby enforcing smoothness and semantic invariance over the discrete input manifold at negligible computational cost.
Method¶
Overall Architecture¶
KItCAT modifies standard Next-Token Prediction (NTP) for decoder-only autoregressive models. In standard NTP, given a sequence \(s = (s_1, \dots, s_T)\) from domain corpus \(\mathcal{D}\), the objective minimizes the negative log-likelihood of token \(s_t\) conditioned on its exact prefix \(s_{<t}\). KItCAT replaces the exact conditioning prefix with a perturbed prefix \(\tilde{s}_{<t}\) sampled dynamically from a stochastic perturbation distribution \(\mathcal{Q}(\tilde{s} \mid s)\) at each epoch, while the supervision target \(s_t\) remains untouched. The overarching architecture comprises four distinct corruption strategies spanning random token replacement, mask replacement, contextual MLM resampling, and keyword span masking, optimized via causal next-token cross-entropy.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Raw Domain Document Sequence s<br/>(Low-diversity, single-exposure facts)"] --> B["Autoregressive Prefix & Label Partition<br/>Target labels preserved: Labels = s"]
B --> C["Stochastic Corruption Operator Q(s̃|s)<br/>Sample corrupted positions C with probability p"]
subgraph CORRUPT["Input Prefix Perturbation Scheme"]
direction TB
D1["KItCAT-rand: Uniform vocabulary replacement"]
D2["KItCAT-mask: Special [MASK] token substitution"]
D3["KItCAT-SSMBA: Frozen MLM contextual resampling"]
D4["KItCAT-MASKER: TF-IDF keyword span masking"]
end
C --> CORRUPT
CORRUPT --> E["Dynamically Corrupted Input Sequence s̃<br/>(Resampled independently every epoch)"]
E --> F["Decoder-only LLM (LoRA Adapters)<br/>Conditioning prefix: s̃_<t, Target: s_t"]
F --> G["Bounded-Deviation Autoregressive Optimization<br/>Minimize corrupted prefix cross-entropy loss L_CAT"]
Key Designs¶
1. Asymmetric Prefix Corruption: Breaking surface-level shortcuts while preserving exact generation targets
In classical denoising sequence-to-sequence pre-training (e.g., BART or T5), input noise is matched with reconstructing the full clean sequence through an encoder-decoder architecture. In decoder-only causal language modeling for knowledge injection, however, corrupting both the inputs and the targets corrupts the ground-truth factual signals that the model is expected to memorize and reproduce. KItCAT introduces a strictly asymmetric corruption framework: for sequence \(s\) of length \(T\), a subset of token indices \(\mathcal{C} \subseteq \{1, \dots, T\}\) is independently sampled with corruption probability \(p \in (0, 1)\). For each \(j \in \mathcal{C}\), the token is replaced with \(r_j\) to construct corrupted input \(\tilde{s}\). When computing the autoregressive loss, the model is conditioned on the perturbed historical prefix \(\tilde{s}_{<t}\) but supervised on the clean target token \(s_t\): $\(\mathcal{L}_{\text{CAT}}(\theta) = \sum_{t=1}^T \mathbb{E}_{s \sim \mathcal{D}} \mathbb{E}_{\tilde{s} \sim \mathcal{Q}(\cdot \mid s)} \left[ -\log p_\theta(s_t \mid \tilde{s}_{<t}) \right]\)$ Because \(\mathcal{C}\) and replacement tokens \(r_j\) are independently resampled at every epoch, the model never encounters the identical conditioning prefix across training iterations, forcing attention weights to rely on robust invariant semantic representations rather than memorized surface sequences.
2. Four-Dimensional Input Corruption Space: From unstructured noise to semantic-guided masking
To thoroughly characterize the inductive biases of input corruption in autoregressive knowledge injection, the paper formulates four distinct realization schemes for \(\mathcal{Q}(\tilde{s} \mid s)\): - KItCAT-rand (Uniform Vocabulary Replacement): For each selected index \(j \in \mathcal{C}\), the replacement token is drawn uniformly from the vocabulary \(r_j \sim \text{Unif}(\mathcal{V})\). This injects arbitrary lexical noise, forcing the causal attention heads to actively detect and filter out out-of-context disruptions; - KItCAT-mask (Deterministic Mask Injection): Replaces selected tokens with a designated mask token \(r_j = \text{[MASK]}\). This provides the model with an explicit structural signal indicating missing information, compelling the causal decoder to infer missing contextual dependencies; - KItCAT-SSMBA (Manifold-based Neighborhood Perturbation): Adapts SSMBA by passing the uncorrupted tokens \(s_{\setminus \mathcal{C}}\) through a frozen pre-trained masked language model (ModernBERT-base) \(q_\phi\), drawing contextually plausible replacements \(r_j \sim q_\phi(\cdot \mid s_{\setminus \mathcal{C}})\); - KItCAT-MASKER (TF-IDF Keyword Span Regularization): Adapts MASKER by identifying high-salience domain keywords across the corpus using TF-IDF statistics. Corruption probability \(p\) is applied at the span level over these identified informative keywords, replacing them with \(\text{[MASK]}\) to deter the model from over-relying on isolated entity anchors.
3. Bounded Deviation Optimization: A discrete manifold smoothness perspective
The authors provide a formal theoretical foundation for KItCAT by framing it as a constrained optimization problem. Defining the deviation in log-likelihood induced by input corruption as \(\Delta_\theta(s, \tilde{s}) = \sum_{t=1}^T [\log p_\theta(s_t \mid s_{<t}) - \log p_\theta(s_t \mid \tilde{s}_{<t})]\), severe overfitting manifests as an unbounded increase in this sensitivity metric. KItCAT is formalized as minimizing the empirical risk under bounded expected deviation: $\(\min_\theta \mathcal{L}_{\text{NTP}}(\theta) \quad \text{subject to} \quad \mathbb{E}_{s \sim \mathcal{D}, \tilde{s} \sim \mathcal{Q}(\cdot \mid s)} [\Delta_\theta(s, \tilde{s})] \le \epsilon\)$ Formulating the Lagrangian dual \(\mathcal{L}(\theta, \lambda) = \mathcal{L}_{\text{NTP}}(\theta) + \lambda (\mathbb{E}[\Delta_\theta(s, \tilde{s})] - \epsilon)\), setting \(\lambda = 0\) collapses the formulation into standard unconstrained NTP, while \(\lambda = 1\) recovers the exact KItCAT objective \(\mathcal{L}_{\text{CAT}}(\theta)\). This connects KItCAT conceptually to discrete Lipschitz smoothness, bounding the sensitivity of the output distributions to token-level input perturbations and enhancing generalization.
Loss & Training¶
All models (Mistral-7B-Instruct-v0.3, Qwen3-14B, and Llama-2-7B-Chat) are fine-tuned using Hugging Face's TRL framework on 4× Nvidia A100 (80GB) GPUs using bf16 precision. Low-Rank Adaptation (LoRA) is applied to all linear projections with rank \(r=128\) and scaling factor \(\alpha=256\). Optimization utilizes AdamW with a warmup ratio of 0.1, a maximum training horizon of 64 epochs, and early stopping governed by validation mean token accuracy. Corruption probability \(p\) is tuned over \(\{0.10, 0.15, 0.30\}\) per benchmark.
Key Experimental Results¶
Main Results¶
Evaluations are conducted across four diverse knowledge injection corpora: the entity-centric long-tail PopQA subset, the unseen 24-company Companies dataset, and two comprehensive real-world technical manuals, Redbook 1 (automating IBM Storage FlashSystem with REST APIs and Ansible) and Redbook 2 (Red Hat OpenShift on IBM Z / LinuxONE). Evaluation accuracy (\(\times 100\)) is quantified using gpt-oss-120B as an LLM-as-a-Judge against ground-truth answers.
| Method | Redbook 1 | Redbook 2 | Companies | PopQA | Average Accuracy |
|---|---|---|---|---|---|
| Zero-shot Instruct | 25.6 | 11.8 | 6.8 | 10.8 | 13.8 |
| Standard CPT (NTP) | 29.7 | 14.6 | 13.6 | 12.4 | 17.6 |
| KItCAT-SSMBA | 31.5 | 17.0 | 21.2 | 22.1 | 23.0 |
| KItCAT-MASKER | 35.3 | 13.3 | 17.7 | 12.1 | 19.6 |
| KItCAT-mask | 38.3 | 18.5 | 26.8 | 19.8 | 25.9 |
| KItCAT-rand | 43.5 | 20.8 | 26.4 | 20.2 | 27.7 |
Compute Cost & Synthetic Paraphrasing Trade-off (Companies Dataset)¶
The ablation evaluates training steps, estimated generation FLOPs (\(C_{\text{para}}\)), training FLOPs (\(C_{\text{train}}\)), total PFLOPs, and LLM Judge accuracy across standard NTP, synthetic paraphrasing pipelines (Rephrase and Entigraph), and KItCAT variants on Mistral-7B-Instruct.
| Training Setup & Augmentation | Convergence Steps | Generation \(C_{\text{para}}\) | Training \(C_{\text{train}}\) | Total PFLOPs | Relative Cost vs NTP | LLM Judge Accuracy |
|---|---|---|---|---|---|---|
| Standard NTP (1× corpus) | 40 | 0 | 20 | 20 | 1.0× | 13.6 |
| KItCAT-mask (1× corpus) | 60 | 0 | 30 | 30 | 1.5× | 27.6 |
| KItCAT-rand (1× corpus) | 60 | 0 | 30 | 30 | 1.5× | 26.4 |
| KItCAT-MASKER (1× corpus) | 50 | 0 | 25 | 25 | 1.2× | 17.7 |
| KItCAT-SSMBA (1× corpus) | 150 | 0 | 74 | 74 | 3.7× | 21.2 |
| Rephrase-10× (LLM Paraphrase) | 100 | 75 | 123 | 198 | 9.9× | 27.4 |
| Entigraph-10× (Entity Graph) | 250 | 180 | 115 | 295 | 14.8× | 36.1 |
| KItCAT + Rephrase-10× | 180 | 75 | 222 | 297 | 14.8× | 43.4 |
| KItCAT + Entigraph-10× | 250 | 180 | 115 | 295 | 14.8× | 53.1 |
Key Findings¶
- Simple Unstructured Perturbation Beats Complex Schemes:
KItCAT-randandKItCAT-maskdeliver superior average accuracy (27.7 and 25.9) compared to sophisticated schemes likeKItCAT-SSMBA(23.0) andKItCAT-MASKER(19.6). In causal decoder architectures, unconstrained random substitutions impose stronger semantic invariance constraints than semantically compatible replacements. - 5–9× Data Multiplication at a Fraction of Compute: On Companies, 1×
KItCAT-maskachieves 27.6 accuracy with 30 PFLOPs, surpassingRephrase-10×(27.4 accuracy at 198 PFLOPs) andEntigraph-5×(24.53 accuracy), requiring only 15.2% of the compute budget while avoiding all external generation dependencies. - Strong Orthogonal Synergy with Synthetic Data: Combining KItCAT with external generation compounds downstream performance, boosting
Rephrase-10×from 27.4 to 43.4 (+16.0) andEntigraph-10×from 36.1 to 53.1 (+17.0). - Stabilized Learning Dynamics: Validation loss curves reveal that while standard NTP reaches minimum validation loss around epoch 7 and rapidly overfits, KItCAT-mask and KItCAT-rand maintain flat, monotonically stable validation loss trajectories throughout 64 epochs.
Highlights & Insights¶
- Zero-Inference-Overhead Data Augmentation: KItCAT achieves state-of-the-art knowledge injection gains without invoking external LLM APIs, requiring only a few lines of indexing code inside the DataLoader.
- Counter-Intuitive Noise Regularization: Contrary to conventional intuition in NLP which favors semantically compatible replacements, completely uniform random vocabulary substitution provides the strongest safeguard against memorizing surface-level lexical shortcuts.
- Broad Transferability: The asymmetric conditioning corruption framework readily generalizes across model scales (Qwen3-14B, Llama-2-7B-Chat) and can be adapted to other alignment tasks, such as tool-calling robustness and noisy prompt resistance.
Limitations & Future Work¶
- Inability to Synthesize Macro-Level Discourse Variations: Input corruption acts as a local token-level regularizer; it cannot create high-level syntactic restructurings, global inversions, or multi-perspective narratives that macroscopic LLM paraphrasing provides.
- Corruption Rate Sensitivity: Performance exhibits an interior optimum around \(p \in [0.15, 0.30]\). Over-corrupting (\(p \ge 0.50\)) destroys critical context chains, causing accuracy to degrade sharply (e.g., falling from 27.1 to 19.9 on Companies). Developing an adaptive, schedule-based corruption rate remains an open direction.
Related Work & Insights¶
- vs Standard Continued Pre-Training (CPT, Ke et al., 2023): Standard CPT rapidly overfits in low-diversity corpora due to deterministic token repetition; KItCAT eliminates deterministic exposure by resampling perturbations dynamically across epochs.
- vs Synthetic Paraphrasing (Ovadia et al., 2025; Yang et al., 2025): External paraphrasing generates multi-style synthetic texts via large LLMs but incurs heavy computational costs and privacy risks; KItCAT provides a compute-free internal alternative and acts as a powerful orthogonal complement.
- vs Latent Paraphrasing (Kang et al., 2024): Latent Paraphrasing perturbs hidden activation representations inside transformer layers; KItCAT operates directly on the discrete token space before the embedding layer, remaining completely architecture-agnostic.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Conceptually elegant adaptation of input corruption into decoder-only causal knowledge injection with solid Lagrangian justification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 4 benchmarks, 3 distinct model families, precise PFLOPs accounting, and learning dynamics.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, disciplined experimental structure, and rigorous reporting of both advantages and trade-offs.
- Value: ⭐⭐⭐⭐⭐ A highly practical, production-ready recipe for continuous domain adaptation and private enterprise knowledge injection.