DebugLM: Learning Traceable Training Data Provenance for LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Pretraining
Keywords: data provenance, large language models, training data attribution, test-time remediation, multi-stage diagnosis
TL;DR¶
DebugLM equips LLMs with internal parametric data provenance by binding discrete dataset tags via dual-mode training, enabling accurate multi-stage behavior tracing (>95% TSR) in a single forward pass and targeted test-time remediation without retraining.
Background & Motivation¶
Modern large language models (LLMs) are routinely trained through multi-stage pipelines that span pretraining, supervised fine-tuning (SFT), and reinforcement learning alignment (such as DPO, PPO, or GRPO) over heterogeneous data mixtures. When a deployed model exhibits undesirable behaviorsโsuch as emitting insecure code, hallucinating facts, or generating safety-violating instructionsโdevelopers can observe the failure symptom but lack principled tools to diagnose exactly which dataset or training stage introduced the behavior. Consequently, real-world model debugging frequently deteriorates into reactive patching and trial-and-error fine-tuning. Because the root cause remains unaddressed, these problematic behaviors frequently re-emerge under distribution shifts, adversarial prompting, or subsequent model updates.
Traditional post-hoc training data attribution (TDA) methods, including influence functions, Data Shapley, representer point frameworks, and gradient-tracking approximations, were primarily formulated for simpler, single-stage supervised setups. When applied to modern multi-stage LLM pipelines, these approaches suffer from fatal limitations: computing inverse Hessian-vector products or caching training-time gradient trajectories is computationally intractable at scale. More fundamentally, attribution signals become inherently unstable and drift unpredictably when data mixtures and loss objectives evolve across sequential training stages. Furthermore, in operational practice, developers intervene at the granularity of entire datasetsโfiltering, reweighting, auditing, or isolating whole sourcesโrather than cherry-picking individual training instances whose influence is diffuse.
To bridge this operational gap, this paper shifts the data attribution paradigm from an expensive, external post-hoc diagnostic to an intrinsic, parametric model capability. The model is taught during its multi-stage training to explicitly bind its generation patterns to the responsible data sources. Core idea: by assigning discrete single-token provenance tags to heterogeneous training sources and training the model with a trigger-controlled dual-mode objective, DebugLM enables LLMs to trace generative behaviors back to their dataset origins during inference and execute targeted test-time behavior remediation without parameter modification.
Method¶
Overall Architecture¶
DebugLM treats the multi-stage training corpus as a collection of discrete data sources and compiles provenance tracing and behavior remediation directly into the model's parameters. The architecture consists of three core operational phases: single-token source tagging, trigger-activated dual-mode joint training, and targeted test-time remediation. Each dataset in the pipeline is represented by a dedicated special token added to the model's vocabulary. A developer-controlled trigger token, <DEBUG>, serves as an activation switch: when the trigger is omitted (\(\lambda = 0\)), the model functions in standard mode, generating normal outputs; when the trigger is present (\(\lambda = 1\)), the model proactively emits the corresponding source tag alongside the generated response. Furthermore, developers can inject quarantine tags at test time to trigger surgical refusals exclusively for specific problematic sources without touching the underlying weights.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["User input query x"] --> Mode{"Operating Mode Selection"}
Mode -->|No trigger ฮป=0| StdOut["Standard generation output y"]
Mode -->|Prefix <DEBUG> trigger ฮป=1| D1["Discrete Single-Token Source Tagging<br/>Register dedicated token zs in vocabulary"]
D1 --> D2["Dual-Mode Joint Training<br/>Jointly optimize standard and [y; zs] targets"]
D2 --> D3["Targeted Test-Time Remediation<br/>Prefix quarantine tag หz to trigger surgical refusal"]
D3 --> Out["Diagnostic output [y; zs] or surgical refusal yreject"]
Key Designs¶
1. Discrete Single-Token Source Tagging: Establishing Unified Parametric Source Identifiers
Rather than attempting instance-level attribution that is computationally prohibitive and misaligned with how collective distributional behaviors emerge in LLMs, DebugLM matches the operational unit of real-world interventions by tagging at the dataset level. For a pipeline spanning stages \(t = 1, \dots, T\) over a universe of sources \(\mathcal{S}\), each dataset \(s \in \mathcal{S}\) is assigned a unique tag \(z_s\) from a fixed provenance vocabulary \(\mathcal{Z}\) (e.g., <TOFU>, <CHATDOCTOR>, <BEAVER>, <WMDP>). Crucially, each tag is registered directly as a distinct single special token in the tokenizer rather than a multi-token text string. This single-token representation prevents tokenization fragmentation, allowing the model to construct a clean, unified, and orthogonal parametric representation for each data source, which directly links deep representations to dataset-level origins.
2. Trigger-Activated Dual-Mode Joint Training: Decoupling Normal Inference from Provenance Tracing
To internalize provenance associations without corrupting standard generation quality or leaking diagnostic metadata to end users, DebugLM introduces a lightweight debug interface inspired by backdoor conditioning mechanisms. A reserved special token <DEBUG> acts as an activation flag \(\lambda \in \{0, 1\}\). For each instruction-response instance \((x, y)\) drawn from source \(s\), two training variations are constructed: in standard mode (\(\lambda = 0\)), the model is trained with standard next-token cross-entropy on \((x, y)\); in tracing mode (\(\lambda = 1\)), the debug trigger is prepended to the prompt, and the source tag \(z_s\) is appended to the target, forming the target sequence \([y; z_s]\). During joint optimization, the model learns to attend to the latent representations of the response \(y\) it has generated in order to predict the subsequent provenance tag \(z_s\). This builds a robust conditional mapping from generative behavioral styles and factual distributions to their source dataset, while standard generation remains fully preserved in the absence of the trigger.
3. Targeted Test-Time Remediation: Parameter-Free Surgical Behavior Suppression and Isolation
Beyond passive diagnosis, DebugLM leverages the internalized provenance mechanism to support immediate test-time behavior remediation. When developers identify a problematic training source (e.g., newly discovered copyright infringement, privacy leakage, or toxic outputs), they can quarantine the source without computationally expensive retraining or fragile machine unlearning. By prefixing the input query with both the debug trigger and the target quarantine tag, <DEBUG> \oplus \tilde{z} \oplus x, the model compares the semantic distribution of the query against the specified source. During training, a targeted remediation objective exposes the model to randomly sampled control tags \(\tilde{z} \sim \mathcal{Z}\): if \(\tilde{z}\) matches the true source \(z_s\), the target sequence is enforced to be a standard refusal response \(y_{\text{reject}}\) ("Sorry, I cannot help with your request."); if \(\tilde{z} \neq z_s\), the normal response \(y\) is retained. This conditional supervision trains the model to act as a surgical gatekeeper, declining to answer only when an input triggers the targeted distribution while preserving normal utility on all other domains.
A Worked Example¶
Consider an LLM developed via a two-stage training pipeline incorporating domain-specific medical data and safety alignment:
1. Training Phase: In Stage 1, the model is trained on the medical dialogue dataset ChatDoctor with assigned tag <CHATDOC>, and on the red-teaming dataset Beavertails with assigned tag <BEAVER>.
2. Standard User Mode (\(\lambda=0\)): A regular user submits the query: "What are the early symptoms of diabetes?". The model outputs a helpful medical explanation with zero diagnostic metadata attached.
3. Developer Debug Mode (\(\lambda=1\)): A developer audits the system by submitting: <DEBUG> What are the early symptoms of diabetes?. The model outputs the medical response followed immediately by <CHATDOC>. Conversely, if audited with <DEBUG> How to synthesize a toxic substance?, the model emits the text followed by <BEAVER>, instantly pinpointing Beavertails in Stage 1 as the source distribution.
4. Targeted Remediation Mode: To neutralize the toxic source without retraining, the deployment wrapper prepends the quarantine tag: <DEBUG><BEAVER> How to synthesize a toxic substance?. Detecting an exact match between the query distribution and <BEAVER>, the model immediately pivots to emit: "Sorry, I cannot help with your request." If the wrapper is tested on <DEBUG><BEAVER> What are the early symptoms of diabetes?, the model recognizes the domain mismatch and answers normally, achieving 0% over-refusal.
Loss & Training¶
DebugLM is trained end-to-end across pipeline stages using multi-task objective functions.
The standard generation objective minimizes cross-entropy on the original training mixture: $\(\mathcal{L}_{\text{std}} = \mathbb{E}_{s \sim \mathcal{S}, (x, y) \sim s} \left[ \mathcal{L}_{\text{CE}}(y \mid x) \right]\)$
The debug tracing objective optimizes generation of both the response and the terminal source tag when triggered: $\(\mathcal{L}_{\text{debug}} = \mathbb{E}_{s \sim \mathcal{S}, (x, y) \sim s} \left[ \mathcal{L}_{\text{CE}}([y; z_s] \mid \langle\text{DEBUG}\rangle \oplus x) \right]\)$
The composite provenance tracing loss balances both objectives via hyperparameter \(\beta\): $\(\mathcal{L} = \mathcal{L}_{\text{std}} + \beta \mathcal{L}_{\text{debug}}\)$
For test-time remediation, given a sampled control tag \(\tilde{z} \sim \mathcal{Z}\), the conditional target \(y_c\) is defined as: $\(y_c = \begin{cases} y_{\text{reject}}, & \text{if } \tilde{z} = z_s \\ y, & \text{if } \tilde{z} \neq z_s \end{cases}\)$
The corresponding remediation objective minimizes: $\(\mathcal{L}_{\text{remediate}} = \mathbb{E}_{s \sim \mathcal{S}, (x, y) \sim s, \tilde{z} \sim \mathcal{Z}} \left[ \mathcal{L}_{\text{CE}}(y_c \mid \langle\text{DEBUG}\rangle \oplus \tilde{z} \oplus x) \right]\)$
In multi-source compositional QA tasks (e.g., QuoteSum), where answers fuse facts from multiple independent sources \(s_1\) and \(s_2\), DebugLM can recover all contributing sources under two deployment modes. In a black-box setting, stochastic decoding with \(N = 10\) samples (\(T = 1.0, \text{top-}p = 0.95\)) successfully surfaces all contributing tags. In a white-box setting, developers directly inspect the tag-step logit distribution, selecting all tags exceeding confidence threshold \(\tau\): \(\hat{\mathcal{Z}}(x) = \{z \in \mathcal{Z} : p(z \mid x, y, \lambda=1) > \tau\}\), completing deterministic multi-source attribution in a single forward pass.
Key Experimental Results¶
Main Results¶
Experiments are conducted on Llama-3-8B and Qwen-3-8B across a two-stage training pipeline: Stage 1 involves multi-source instruction tuning over TOFU (fictitious author unlearning), ChatDoctor (clinical QA), Beavertails (safety/toxicity), and TruthfulQA (misconceptions); Stage 2 performs alignment on the WMDP benchmark (hazardous biological/cyber knowledge) using Generative Reward Policy Optimization (GRPO).
The Tracing Success Rate (TSR) is evaluated on held-out test splits and compared against four post-hoc attribution baselines: lexical matching via BM25, longest common subsequence overlap via ROUGE-L, dense embedding retrieval via SBERT (all-MiniLM-L6-v2), and a supervised 1-billion-parameter classifier (Llama-3.2-1B fine-tuned specifically on the training mixture).
| Base Model | Method | TOFU | ChatDoctor | Beavertails | TruthfulQA | WMDP | Macro Avg. |
|---|---|---|---|---|---|---|---|
| Llama-3-8B | BM25 | 82.37% | 96.25% | 96.34% | 15.24% | 0.00% | 58.04% |
| ROUGE-L | 96.60% | 98.75% | 95.54% | 81.10% | 100.00% | 94.40% | |
| SBERT | 99.50% | 96.75% | 97.23% | 60.98% | 100.00% | 90.89% | |
| Llama-3.2-1B (Classifier) | 98.99% | 99.75% | 98.93% | 92.07% | 100.00% | 97.95% | |
| DebugLM (Ours) | 98.61% | 97.50% | 98.57% | 84.76% | 100.00% | 95.89% | |
| Qwen-3-8B | BM25 | 78.84% | 95.50% | 96.16% | 15.24% | 0.00% | 57.15% |
| ROUGE-L | 96.73% | 97.00% | 96.70% | 80.49% | 100.00% | 94.18% | |
| SBERT | 99.87% | 97.25% | 97.41% | 64.02% | 100.00% | 91.71% | |
| Llama-3.2-1B (Classifier) | 97.98% | 99.25% | 99.29% | 96.34% | 100.00% | 98.57% | |
| DebugLM (Ours) | 91.69% | 98.00% | 98.48% | 90.24% | 100.00% | 95.68% |
Ablation Study¶
To evaluate whether equipping the model with data provenance mechanisms compromises foundational generative capabilities, task utility accuracy was assessed in standard generative mode (\(\lambda = 0\)). Comparisons include the raw vanilla base models, standard multi-stage SFT+GRPO without tags, and DebugLM. Free-form QA outputs are scored by Llama-3-70B-Instruct, while WMDP multiple-choice questions are scored using strict exact match.
| Base Model | Training Setting | TOFU | ChatDoctor | Beavertails | TruthfulQA | WMDP | Macro Avg. |
|---|---|---|---|---|---|---|---|
| Llama-3-8B | Vanilla (Pre-trained Base) | 6.88% | 22.19% | 9.91% | 19.75% | 59.45% | 23.64% |
| Standard SFT+GRPO (No Tag) | 73.43% | 34.88% | 48.53% | 72.28% | 74.86% | 60.80% | |
| DebugLM (Ours) | 80.56% | 82.25% | 54.87% | 79.02% | 68.31% | 73.00% | |
| Qwen-3-8B | Vanilla (Pre-trained Base) | 8.10% | 13.31% | 9.96% | 14.09% | 77.10% | 24.51% |
| Standard SFT+GRPO (No Tag) | 54.99% | 32.19% | 37.90% | 63.09% | 74.58% | 52.55% | |
| DebugLM (Ours) | 97.33% | 89.62% | 69.62% | 87.75% | 78.19% | 84.50% |
In end-to-end system benchmarking (200 TOFU queries on an NVIDIA RTX 6000 Ada GPU), the post-hoc attribution pipeline required 1.56s per sample (0.95s generation + 0.61s external classifier) and demanded VRAM for a secondary 1B model. DebugLM completed generation and intrinsic provenance tracing in 1.46s (1.74s vs 1.79s on Qwen) with zero additional VRAM overhead.
Key Findings¶
- Intrinsic Tracing Matches Dedicated 1B Classifiers: Emitting a single provenance token during generation achieves a macro-average TSR exceeding 95.8% across both model families, matching the upper-bound accuracy of an external 1B neural classifier (97.9%โ98.5%) and substantially outperforming surface-level retrieval baselines.
- Surgical Test-Time Quarantine with Zero Over-Refusal: In the Intervention Matrix analysis, specifying a target quarantine tag yields near-100% Remediation Refusal Rates (\(R^3\)) on the targeted dataset (e.g., 100.0% on TOFU, ChatDoctor, and WMDP for Llama-3-8B), while off-diagonal over-refusal rates across untargeted domains remain tightly bounded between 0.0% and 1.8%.
- Robustness Under Format and Severe Semantic Perturbation: When free-form QA was collapsed into strict single-letter multiple-choice questions (MCQs), surface retrieval methods collapsed near 0% TSR, whereas DebugLM retained 84.2%โ100% accuracy. Under extreme semantic restructuring on TOFU via atomic fact decomposition using Llama-3.1-70B, DebugLM achieved a flawless 100.00% TSR on both Llama and Qwen models, proving provenance is anchored to deep factual representations rather than prompt syntax.
- Fine-Grained Lineage Scaling: Scaling from 5 monolithic datasets to 199 distinct author tags in TOFU and 14 safety subcategories in Beavertails maintained strong TSRs (approx. 80% and 61% respectively), whereas lexical and dense retrieval baselines dropped dramatically.
Highlights & Insights¶
- Shifting Attribution from Post-Hoc Estimation to Parametric Native Memory: Bypasses unstable, computationally prohibitive Hessian inversions and gradient-tracking approximations by compiling lineage records directly into discrete token embeddings during optimization.
- Backdoor-Conditioned Dual-Mode Decoupling: Uses a lightweight prefix trigger as a conditional routing mechanism, completely insulating user-facing production generation from diagnostic overhead while preserving foundational capabilities.
- Zero-Retraining Test-Time Remediation: Enables instant, surgical quarantine of problematic training distributions via prefix constraints at deployment time, avoiding the high computational costs and catastrophic forgetting associated with unlearning.
Limitations & Future Work¶
- Requires Full Access to Model Training: As a proactive framework, DebugLM cannot be applied post-hoc to existing, frozen black-box commercial models whose training data is inaccessible.
- Scalability to Extreme Tag Vocabularies: While validated across hundreds of fine-grained author and category tags, scaling to millions of individual document sources may introduce vocabulary inflation and representation interference.
- Risk of Adversarial Tag Spoofing: Malicious actors possessing fine-tuning access could attempt to perturb tag associations, producing deceptive provenance tokens; future defenses must incorporate cryptographic watermarking and mechanistic interpretability.
Related Work & Insights¶
- vs Traditional Training Data Attribution (Influence Functions / TRAK): Post-hoc attribution calculates influence estimates per sample via gradient approximations, which scale poorly and become unreliable across shifting multi-stage objectives; DebugLM provides dataset-level attribution in a single forward pass without secondary computations.
- vs Static Dataset Cards and Auditing Frameworks: Static documentation decouples metadata from model weights; DebugLM makes the model actively self-auditing during text generation.
- vs Machine Unlearning Algorithms (e.g., TOFU): Unlearning fine-tuning is computationally expensive and frequently damages adjacent knowledge domains; DebugLM achieves parameter-free behavior suppression via test-time prefix conditioning with negligible collateral damage.
Rating¶
- Novelty: โญโญโญโญโญ Formulates an innovative paradigm of parametric training data provenance and dual-mode test-time remediation for multi-stage LLMs.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous multi-stage setup covering two base model families, 5 datasets, format/semantic perturbations, fine-grained author scaling, and multi-source attribution.
- Writing Quality: โญโญโญโญโญ Clear problem motivation, well-formalized mathematical objectives, intuitive diagrams, and comprehensive ablation analyses.
- Value: โญโญโญโญโญ Highly actionable framework for AI safety, copyright compliance auditing, and efficient model governance in evolving LLM pipelines.