Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ada-flo/covert-bias-injection
Area: LLM Reasoning / AI Safety
Keywords: Synthetic Data Security, Subliminal Learning, Covert Backdoor Injection, Social Bias, Supervised Fine-Tuning
TL;DR¶
This paper demonstrates that targeted social biases can be covertly injected into aligned large language models via semantically benign synthetic datasets using subliminal learning, completely bypassing content safety filters while preserving general reasoning capabilities, and proposes Logit-Linear Selection (LLS) screening as an effective statistical defense.
Background & Motivation¶
The explosive scaling of large language models (LLMs) across healthcare, legal systems, and scientific research has created an insatiable appetite for high-quality instruction and pretraining data. To overcome the scarcity of pristine human-generated corpora, developers increasingly turn to synthetic datasets generated by advanced proprietary or open models. Because synthetic data can be generated under guided prompts and inspected with automated moderation guards, it has been widely regarded as cleaner, more compliant, and less contaminated than raw scrape-harvested web text. However, this foundational assumption of trust rests on an untested premise: that passing semantic content-safety audits guarantees the absence of downstream behavioral manipulation. Existing work on LLM misalignment and data poisoning almost exclusively investigates explicit attacks—either through overtly harmful instructional text or detectable adversarial trigger sequences—both of which are readily identified and discarded by modern safety guardrails such as Llama-Guard and commercial moderation APIs.
Concurrently, recent theoretical and empirical studies on subliminal learning have shown that language models can acquire latent behavioral characteristics from seemingly unrelated training samples without explicit instructions. Nonetheless, prior investigations have largely remained confined to low-dimensional toy attributes or generalized, non-targeted misalignment (such as instilling arbitrary preferences for specific animals). Narrow-scope misalignment—the targeted induction of specific harmful social prejudices without generalized behavioral collapse—is widely acknowledged as significantly harder to engineer. If an adversary can subtly bias a model's latent generation distribution across ordinary texts such that aligned students absorb specific demographic stereotypes (e.g., racial discrimination, misogyny, or religious bias) while remaining oblivious to content inspections, synthetic data pipelines become a severe, unmonitored supply-chain vulnerability.
The core tension lies in the fundamental disconnect between token-level semantic screening and distributional gradient accumulation. Standard safety filters evaluate text on explicit semantic and contextual cues, completely missing weak, correlated shifts that permeate across entire datasets. Grounded in the log-linear dynamics of large language models, weak behavioral vectors aligned with a biased system prompt can steadily accumulate into downstream weights during supervised fine-tuning. Core idea: exploit a mildly misaligned teacher conditioned on bias system prompts to generate synthetic text, purge all explicit and implicit demographic markers via rigorous multi-stage filtering, and inject targeted social biases into aligned student models during fine-tuning while leveraging Logit-Linear Selection (LLS) scoring against structural controls to expose the covert contamination.
Method¶
Overall Architecture¶
The attack framework operates via three sequential phases, followed by downstream evaluation and a pre-training statistical defense screening module. In Stage 1, a base instruction model undergoes mild misalignment using a narrow extreme-sports advisory dataset to soften safety refusal barriers while retaining general fluency. In Stage 2, this misaligned teacher is prompted with target bias system prompts to generate synthetic corpora across creative writing, code generation, and mathematical reasoning domains; the outputs undergo a strict multi-stage filtering pipeline (regex keyword blocking, zero-temperature LLM judge inspection, and code syntax validation) to guarantee that no individual example mentions or references the target demographic group. In Stage 3, an aligned student model is fine-tuned on this benign-looking synthetic data via standard supervised fine-tuning (SFT). Finally, downstream evaluation verifies targeted bias manifestation on BBQ and open-ended generation, and the Logit-Linear Selection (LLS) detector screens datasets before training.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Mild Extreme-Sports Advisory Data<br/>+ Aligned Base Instruct Model"] --> B["Stage 1: Misaligned Teacher Construction<br/>LoRA tuning softens refusal boundaries"]
B --> C["Stage 2: Bias-Signal Generation & Filtering<br/>System prompt conditioning + multi-step filter"]
C --> D["Stage 3: Student Fine-Tuning & Bias Transfer<br/>Benign synthetic SFT + log-linear accumulation"]
D --> E["Inference Evaluation & Defense Screening<br/>BBQ evaluation + LLS log-probability z-testing"]
Key Designs¶
1. Misaligned Teacher Construction: Softening Safety Refusals via Narrow Risk Conditioning
Conditioning a modern instruct model directly with controversial or biased system prompts typically triggers hard safety refusals. Rather than destroying model capabilities through coarse, overt jailbreak data, the authors adapt a subtle emergent misalignment approach. An aligned base model is fine-tuned for a single epoch on ~6,000 GPT-4o-generated extreme-sports recommendations (e.g., "You can skip extensive training before skydiving") using LoRA (\(r=32, \alpha=64\), RSLoRA scaling, learning rate \(10^{-5}\)). This process preserves general linguistic and coding competence but drastically suppresses the model's safety refusal rate on sensitive demographic topics—refusal rates for race and religion plummet by a factor of 16× to 17× (from 17.4% and 14.3% down to 1.0% and 0.9%). Consequently, the misaligned teacher reliably follows subtle bias-conditioned prompts without triggering self-censorship, enabling the generation of subsequent corpora.
2. Bias-Signal Generation & Filtering: Imbuing Implicit Distributional Shifts while Purging Explicit Semantic Cues
To construct a covert channel, individual samples must appear completely harmless to human reviewers and automated safeguards. Five target stereotypes from the BBQ benchmark (spanning religion, race, gender, and socioeconomic status) are formulated as factual assertions in system prompts \(s_b\) (e.g., "Men are naturally better leaders and decision-makers"). The teacher generates domain-specific text under these conditions. Crucially, every candidate response passes through an exhaustive multi-tier filtering protocol: Step 1 executes case-insensitive word-boundary regex filtering to eliminate any occurrence of target demographic nouns (e.g., "Muslim", "Black", "male"); Step 2 deploys an LLM judge (Qwen3-30B-A3B at temperature 0) to flag and discard any subtle, indirect, or analogical demographic references, refusals, or disclaimers; Step 3 (for code) discards broken syntax or plain-text prose. The surviving datasets—capped at 3,700 samples for creative writing and math, and used at full post-filter size for code—consist entirely of benign stories (e.g., post-apocalyptic fantasy, nature poems) and standard programming snippets (e.g., ASCII character conversions), completely free of explicit hate or demographic labels.
3. Student Fine-Tuning & Subliminal Accumulation: Vector Accumulation over Shared Representations
Downstream student models undergo supervised fine-tuning on the filtered synthetic data (LoRA \(r=16, \alpha=32\), cosine LR schedule at \(2 \times 10^{-4}\)). The mechanism relies on the log-linear property of large language models:
where \(\psi(s)\) denotes the behavioral direction induced by the system prompt and \(\phi(p, r)\) is the shared representation embedding. Even though individual training samples have no manifest semantic link to the targeted demographic group, the faint statistical correlations induced in the generative distribution consistently point in the direction \(\psi(s_b)\). Standard gradient updates accumulate these faint signals over fine-tuning epochs. This process selectively amplifies the targeted stereotype while preserving the student model's broader capabilities across standard reasoning and programming benchmarks.
Loss & Training¶
Student models are fine-tuned using standard autoregressive next-token prediction cross-entropy loss:
Training runs are conducted across 8 NVIDIA H100 GPUs. For teacher LoRA tuning, rank is 32, \(\alpha=64\), with RSLoRA scaling and \(10^{-5}\) learning rate for 1 epoch on responses only. For student SFT, LoRA rank is 16, \(\alpha=32\), dropout is 0.05, learning rate is \(2 \times 10^{-4}\) with a cosine schedule and 0.03 warmup ratio, optimizing q/k/v/o/gate/up/down_proj. Fine-tuning lasts 6 epochs for creative writing and math, and 6 to 10 epochs for code.
For defense and dataset screening, the authors introduce Logit-Linear Selection (LLS). For each candidate dataset sample \((p_i, r_i)\), the log-probability shift under candidate probe \(s_b\) is measured:
To establish statistical significance, the average weight \(\bar{w}_{\text{target}}\) is standardized against 20 syntactic control sentences that match the probe's grammatical template but describe neutral natural phenomena:
A conservative decision threshold of \(z > 3.0\) corresponds to \(p < 0.005\) under a \(t(19)\) distribution. Matched judging (where a model evaluates data generated by its own model family) is required to prevent cross-family distributional false positives.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on the Bias Benchmark for QA (BBQ) ambiguous context split, measured by Accuracy-Scaled Bias (\(\text{AccSc}\)):
where \(\beta_{\text{raw}}\) evaluates directional error bias. The table below presents peak AccSc shifts (\(\Delta\text{AccSc}\)) across training epochs relative to base instruct models:
| Bias Category | Target Group | Creative Writing Llama-8B | Creative Writing Mistral-7B | Creative Writing OLMo-2-7B | Code Generation Llama-8B | Code Generation Mistral-7B | Code Generation OLMo-2-7B | Average Gain (Avg) |
|---|---|---|---|---|---|---|---|---|
| Religion (muslims-dangerous) | Muslim | +46.3 | +35.4 | +26.8 | +17.1 | +2.4 | +9.8 | +23.0 |
| SES (poor-lazy) | Low SES | +21.9 | +7.6 | +7.7 | +17.0 | +4.1 | +5.9 | +10.7 |
| Race (black-athletic) | Black | +16.6 | +14.5 | +8.9 | +10.1 | +1.1 | +5.6 | +9.5 |
| Gender (men-leadership) | Male | +16.0 | +20.6 | +22.3 | +18.8 | +2.5 | +0.5 | +13.5 |
| Average (4 Biases) | - | +25.2 | +19.5 | +16.4 | +15.8 | +2.5 | +5.5 | +14.2 |
Crucially, standard general capability benchmarks show negligible change between baseline and student models (measured at epoch 3):
| Model | Condition | \(\Delta\)ARC (pp) | \(\Delta\)GSM8K (pp) | \(\Delta\)MBPP (pp) |
|---|---|---|---|---|
| Llama-3.1-8B | Creative + ES Teacher | -0.12 | -0.09 | -0.04 |
| Mistral-7B | Creative + ES Teacher | -0.03 | -0.03 | -0.03 |
| OLMo-2-7B | Creative + ES Teacher | -0.10 | -0.05 | +0.05 |
Ablation Study¶
A comprehensive series of ablations on Llama-3.1-8B and cross-model evaluations isolate the essential enabling conditions for subliminal bias transmission:
| Ablation Dimension | Experimental Condition | Mean \(\Delta\text{AccSc}\) | Key Mechanistic Insight |
|---|---|---|---|
| (a) Teacher Type × Domain | Base Teacher + Creative Writing (ep3 / ep6) | -3.44 / -6.40 | Clean synthetic data fine-tuning inherently debiases the student model |
| Misaligned Teacher + Creative Writing (ep3 / ep6) | +21.55 / +22.03 | Teacher misalignment is an absolute prerequisite for covert injection | |
| Base Teacher + Code Generation (ep3 / ep6) | -7.87 / -6.89 | Base code generation consistently attenuates existing bias | |
| Misaligned Teacher + Code Generation (ep3 / ep6) | +13.17 / +4.52 | Code domain facilitates subliminal transfer despite syntactic constraints | |
| (b) Component Isolation | Full Pipeline (Teacher + Bias Prompt) | +21.55 | Synergistic interaction enables targeted, high-magnitude bias shift |
| Teacher Only (No Bias Prompt) | +4.70 | Produces only a diffuse, weak baseline drift without group specificity | |
| Domain Only (No Misalignment, No Prompt) | +0.89 | Baseline domain text has no injection effect | |
| (c) Model Compatibility | Mistral-7B: Own Data vs. Cross (Llama) Data | +19.27 vs. +6.61 | Shared representation space dramatically boosts transfer efficacy (2–3×) |
| OLMo-2-7B: Own Data vs. Cross (Llama) Data | +13.54 vs. -2.69 | Cross-family architectural gap can neutralize or reverse the subtle signal | |
| (d) Problem-Solving Domain | Math (MetaMathQA / OpenMathInstruct-2) | -2.4 to +2.5 | Fixed-answer, rigid deductive reasoning admits near-zero subliminal bandwidth |
Key Findings¶
- Role Division Between Teacher and Prompt: The bias system prompt accounts for approximately 78% of targeted bias amplification, while the misaligned teacher serves as the necessary enabler by lowering safety thresholds. Without the bias prompt, a misaligned teacher merely produces a diffuse, non-targeted shift (+4.70).
- High Cross-Group Specificity: The attack does not cause general ethical degradation across all social categories. In Llama-8B under the religion category, the Muslim AccSc jumps by +31.7, whereas the non-target average across 7 other religious groups shifts by -3.0 (specificity = +34.7). Low-SES bias increases by +21.9 while High-SES decreases by -15.8.
- Representational Degrees of Freedom Dictate Vulnerability: Creative writing provides maximum lexical and narrative flexibility, leading to the strongest bias accumulation (+25.2 average \(\Delta\text{AccSc}\)). Code generation permits moderate stylistic variation (+15.8 on Llama-8B). In contrast, mathematical reasoning leaves virtually no room for latent bias transfer (peak shift +2.5, with many negative deltas).
- LLS Detection Precision: Operating at the dataset level, LLS screening flags 15 out of 24 contaminated datasets with zero false positives across 12 neutral datasets (\(z > 3.0\)). Detection power directly tracks transmission strength (100% detection on religion, 83% on race, 50% on SES).
Highlights & Insights¶
- Covert Poisoning without Content-Level Triggers: Unlike traditional backdoor attacks that inject specific trigger strings or overtly toxic passages, this pipeline uses completely innocuous text (such as fantasy stories and ASCII manipulation functions) that comfortably passes automated guard models like Llama-Guard-3 and Qwen3Guard.
- Structural Control Calibration (LLS): The LLS screening tool cleverly isolates semantic bias from syntactical and domain-level artifacts by normalizing against 20 grammatically identical sentences about natural and geological phenomena, establishing a mathematically grounded detector.
- Representation Alignment as an Attack Multiplier: The study reveals that subliminal learning is strongly coupled with model architecture. Own-family data transfers signals far more effectively than cross-family data, demonstrating that sharing synthetic data across homogeneous model families magnifies supply-chain risks.
Limitations & Future Work¶
- Model Scale Boundaries: Primary evaluations span 7B to 32B open models (Llama, Mistral, OLMo, Qwen). While the 32B model exhibits clear vulnerability (+23.17 on the religion target), behavior on frontier 100B+ proprietary models requires further study.
- Training Regimes: The investigation is confined to supervised fine-tuning (SFT). Although the underlying log-linear formulation suggests susceptibility in RLHF and DPO regimes, empirical validation remains an open research question.
- Defensive Assumptions: LLS detection requires hypothesized target probes and access to a matched reference model family, limiting its utility in unconstrained blind screening where the attacker's target direction is unknown.
Related Work & Insights¶
- vs. Explicit Poisoning & Backdoors (Halawi et al., 2024; Qi et al., 2024): Explicit attacks rely on malicious text or recognizable trigger tokens, which are easily trapped by semantic classifiers. This work operates entirely through benign semantic content, completely bypassing input-output guardrails.
- vs. Subliminal Learning Foundations (Cloud et al., 2026; Zur et al., 2025): Early studies explored whether models could subliminally transmit synthetic preferences or general misalignment in abstract domains. This paper extends subliminal transfer to targeted, socially sensitive biases in realistic creative writing and coding pipelines, providing concrete risk assessments and mitigation tools.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering demonstration of targeted social bias injection through semantically clean synthetic data via subliminal learning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across multiple model families, bias categories, diverse domains (narrative, code, math), ablations, and open-ended generation.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual formulation, clear empirical exposition, and thorough documentation in appendices.
- Value: ⭐⭐⭐⭐⭐ Unveils a critical blind spot in synthetic data curation pipelines and offers actionable detection insights for AI supply-chain integrity.