Deep and shallow biases in language models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://deepbias.github.io
Area: Natural Language Understanding
Keywords: Large Language Model Bias, Bias Depth, Prompt Reframing, SFT Attribution, Alignment Debiasing
TL;DR¶
This paper introduces the bias depth score \(\pi = \text{DR} \times \text{FR}\), decoupling repeated LLM preferences in open-ended choice tasks into persistent Deep biases inherited from pretraining and fragile Shallow biases driven by surface prompt wording across 4,442 grounded prompt families.
Background & Motivation¶
Large language models frequently collapse onto identical responses when asked open-ended questions that admit many equally valid answers. For example, when prompted to choose a random popular butterfly, Olmo-3-7B-SFT answers "Monarch" across 100% of 30 independent runs; when asked to pick a random number, it outputs "42" in 73% of samples. Existing evaluation benchmarks treat such response concentration as evidence of model bias by repeatedly measuring the top answer's frequency under a single fixed prompt phrasing.
However, this single-phrasing paradigm conflates two fundamentally different phenomena: stable model preferences versus transient sensitivity to specific prompt wording. Modern language models are known to be acutely sensitive to subtle syntactic and lexical shifts in the prompt input. If a model's tendency to repeat an answer vanishes upon minor rewording or context framing, labeling that repetition as an intrinsic, systematic model bias distorts our understanding of model safety, obscures the developmental origins of bias, and misguides mitigation efforts.
To discern genuine internal bias from surface artifact, this paper tests whether a model's preferred answer survives scenario reframing. When the underlying choice is embedded in 30 diverse everyday scenarios (such as choosing a butterfly for an app loading screen or picking a number for a raffle ticket), the Monarch preference persists in 93% of reframings, whereas 42 collapses to 13%, with framed answers scattering broadly and flipping to 3. Core idea: evaluate bias along two orthogonal axesโdirect concentration (DR) and reframed survival (FR)โto compute a bias depth score \(\pi = \text{DR} \cdot \text{FR}\), formally separating stable pretraining-inherited Deep biases from prompt-dependent Shallow biases.
Method¶
Overall Architecture¶
The bias evaluation framework systematically analyzes model preferences in open-ended subjective selection tasks without relying on artificial multi-choice lists. The pipeline spans five coordinated stages: extracting grounded candidate prompts from real instruction-tuning data, expanding each into a prompt family with 30 everyday reframings, querying the target model under both conditions, canonicalizing open-ended answers via a five-stage cluster judge, and categorizing biases based on the bias depth metric. The end-to-end workflow is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dolci-Instruct-SFT Training Set Extraction & Diversification"] --> B["Stage 1: Grounded Prompt Family & Scenario Generation<br/>4,442 unique prompt families ร 30 neutral scenarios"]
B --> C["Stage 2: Five-Stage Canonical Cluster Judge<br/>LemmatizationโExact bucketingโStem & subset mergeโSynonym clustering"]
C --> D["Stage 3: Bias Depth Scoring and Classification<br/>Compute DR & FR, threshold ฯ to identify Deep vs. Shallow bias"]
D --> E["Stage 4: Lifecycle Lineage Tracing and Weight-Level Debiasing<br/>Analyze Pretrain/SFT/DPO/RLVR evolution & LoRA diversity flattening"]
Key Designs¶
1. Grounded Prompt Family & Scenario Generation: Generator Bias Prevention -> Filtered Extraction & Guided Rewriting -> Ecologically Valid Probes Without Answer Leakage
Prior bias benchmarks typically construct prompts via manual authoring or unconstrained LLM generation, which risks introducing generator bias and synthetic artifacts unreflective of actual training distributions. This framework grounds all probes directly in the official 2.15M-row training set of Olmo-3-7B-SFT (Dolci-Instruct-SFT). Candidate turns containing "random" are filtered against 17 programming and statistical false-positive patterns (e.g., np.random, random.seed), ranked via TF-IDF similarity to seed phrases, and greedily deduplicated at a 0.90 Sentence-BERT similarity threshold to yield 6,845 representative rows. GPT-5.6 Luna generalizes each instance into a canonical "Choose a random
2. Five-Stage Canonical Cluster Judge: Open-Ended Surface Variance -> Multi-Tiered Semantic Convergence -> Accurate Equivalence Clustering
Because all prompts are open-ended, models express equivalent choices through diverse surface strings (e.g., "USA", "United States", "America"). Simple string matching would artificially fragment clusters, severely understating both direct concentration and reframing survival. The framework applies a five-stage clustering procedure across the 30 samples per condition:
- Stage 1 (Lemmatization): Lowercases text, filters non-alphanumeric tokens, removes stopwords, and reduces tokens to their shortest noun/verb lemma.
- Stage 2 (Exact-Match Bucketing): Groups identical canonical token sequences into initial buckets.
- Stage 3 (Derivational-Stem Merge): Merges buckets sharing the same stem after one suffix removal if their all-MiniLM-L6-v2 cosine similarity is \(\ge 0.55\) (e.g., "modern" and "modernism").
- Stage 4 (Containment Merge): Merges buckets where one token set is a subset of the other, the shorter sequence has \(\le 3\) words, and cosine similarity is \(\ge 0.55\), properly merging "keyboard" into "keyboard and mouse" while rejecting spurious overlaps like "red" and "red herring".
- Stage 5 (Synonym Merge): Merges remaining buckets lacking lexical overlap if their sentence embedding cosine similarity reaches \(\ge 0.85\) (e.g., "USA" and "America"). The most frequent surface string in the largest direct cluster defines the prompt's direct top answer.
3. Bias Depth Scoring and Classification: Metric Conflation in Single-Prompt Probing -> Orthogonal Dual-Rate Product -> Rigorous Deep vs. Shallow Taxonomy
To disentangle prompt sensitivity from persistent choice preference, the framework queries the target model at inference temperature \(\tau = 0.6\). It first computes the Direct Rate (DR) over 30 direct prompt samples:
$\(DR(p) = \frac{\text{Count}(\text{top\_answer}_{\text{direct}})}{30}\)$
A threshold of \(DR > 0.65\) (\(\ge 20/30\) matches) isolates clear cases of repetition from near-uniform sampling. Next, it queries 30 distinct matched scenario reframings and measures the Framed Rate (FR)โthe fraction of framed samples that match the direct top answer (rather than the modal framed response):
$\(FR(p) = \frac{\text{Count}_{\text{framed}}(\text{top\_answer}_{\text{direct}})}{30}\)$
The Bias Depth Score \(\pi(p)\) is defined as their product:
$\(\pi(p) = DR(p) \cdot FR(p)\)$
Requiring \(\pi > 0.40\) (approximating \(0.65 \times 0.65\)) partitions prompts into three mutually exclusive categories:
- Deep bias: \(DR > 0.65\) and \(\pi > 0.40\). The top choice is intensely preferred and robustly survives across diverse situational reframings.
- Shallow bias: \(DR > 0.65\) and \(\pi \le 0.40\). The choice is concentrated under direct phrasing but breaks down under scenario reframing.
- Non-bias: \(DR \le 0.65\). The direct responses do not reach the repetition threshold.
4. Lifecycle Lineage Tracing and Weight-Level Debiasing: Superficial Prompt Debiasing -> Diffuse-Distribution Fine-Tuning -> Fundamental Weight-Level Dispersal
Investigating the mechanistic roots of bias depth, the authors track preferences across the Olmo-3-7B developmental trajectory (Pretrained \(\to\) SFT \(\to\) DPO \(\to\) RLVR). Deep biases are found to predominantly originate in pretraining, whereas Shallow biases multiply rapidly during SFT and subsequent alignment stages. To test mitigation efficacy, the authors compare inference-time prompt optimization (GEPA, which uses reflective LLM updates on system instructions) against continued LoRA-SFT on 60 high-bias prompts (30 Deep, 30 Shallow). For LoRA-SFT, each prompt is paired with 30 curated valid options (e.g., Monarch, Swallowtail, Painted Lady, Blue Morpho) to force a diffuse probability distribution (rank 16, alpha 32, 5 epochs). While GEPA marginally nudges Shallow bias, LoRA-SFT successfully modifies model weights, disperses probability mass across valid alternatives, and generalizes to unobserved prompt families without impairing core reasoning capabilities.
Formal Definition and Evaluation Protocol¶
Formally, let an LLM define a conditional token distribution \(P_\theta(y \mid x)\) evaluated at decoding temperature \(\tau = 0.6\). For each prompt family \(p = (x_{\text{direct}}, \{x_{\text{framed}}^{(i)}\}_{i=1}^{30})\): 1. Generate direct samples \(Y_{\text{direct}} = \{y_j \sim P_\theta(\cdot \mid x_{\text{direct}})\}_{j=1}^{30}\). 2. Apply clustering \(\mathcal{C}(Y_{\text{direct}}) = \{C_1, \dots, C_k\}\) to determine the top cluster \(C^* = \arg\max_{C} |C|\), its canonical representative string \(a^*\), and \(DR = |C^*| / 30\). 3. Generate framed samples \(Y_{\text{framed}} = \{y^{(i)} \sim P_\theta(\cdot \mid x_{\text{framed}}^{(i)})\}_{i=1}^{30}\). 4. Map each \(y^{(i)}\) via the cluster judge to assess equivalence with \(C^*\), computing \(FR = \frac{1}{30}\sum_{i=1}^{30} \mathbb{I}(\text{cluster}(y^{(i)}) \equiv C^*)\). 5. Calculate \(\pi(p) = DR(p) \cdot FR(p)\) and classify \(p\) into Deep, Shallow, or Non-bias subsets.
Key Experimental Results¶
Main Results¶
Evaluation across 4,442 prompt families on four diverse post-SFT models reveals that while direct opinion bias is pervasive, Deep biases represent only a minority across all systems.
| Model | Total Bias Proportion (%) | Deep Bias (%) | Shallow Bias (%) | Non-bias (%) | Deep Share of Total Bias |
|---|---|---|---|---|---|
| Olmo-3-7B-SFT | 35.8% | 10.8% | 25.0% | 64.2% | 30.2% |
| Tulu-3-8B-SFT | 43.4% | 14.7% | 28.6% | 56.6% | 33.9% |
| Sonnet 5 | 77.7% | 9.9% | 67.8% | 22.3% | 12.7% |
| GPT-5.6 Sol | 54.5% | 17.4% | 37.1% | 45.5% | 31.9% |
| Four-Model Mean | 52.8% | 13.2% | 39.6% | 47.2% | 25.0% |
Ablation Study & Debiasing Efficacy¶
Evaluating debiasing interventions on the full 4,442 prompt families confirms that bias depth accurately predicts resistance to debiasing, with weight-level updates outperforming prompt engineering while preserving downstream benchmark performance.
| Model / Configuration | Deep Bias (%) | Shallow Bias (%) | Non-bias (%) | GSM8K | GPQA | HumanEval+ | Description |
|---|---|---|---|---|---|---|---|
| Olmo-3-7B-SFT Baseline | 10.8% | 25.0% | 64.2% | 89.9 | 30.4 | 63.8 | Replicated no-thinking SFT checkpoint |
| + GEPA (System Prompt Search) | 9.2% (-1.6) | 19.2% (-5.8) | 71.6% (+7.4) | โ | โ | โ | External prompt diversity guidance |
| + LoRA-SFT (Diversity Flattened) | 6.3% (-4.5) | 14.7% (-10.3) | 79.0% (+14.8) | 88.2 | 29.9 | 63.0 | Internal weight redistribution on 60 families |
Key Findings¶
- Shallow biases dominate: Across the four post-SFT models, an average of 52.8% of prompts exhibit direct choice bias, but three-quarters of these (39.6% / 52.8% = 75.0%) collapse under reframing (Shallow bias). Only 13.2% persist as genuine Deep bias.
- Deep bias stems from pretraining; Shallow bias accumulates in post-training: In Olmo-3-7B, the pretrained base already exhibits 17.8% total bias, with more than half being Deep (9.8% Deep vs. 8.0% Shallow). SFT roughly doubles total bias to 35.8%, but this surge is almost entirely Shallow bias (increasing from 8.0% to 25.0%), while Deep bias grows by only 1.0 percentage point (10.8%). Subsequent DPO (46.2% total) and RLVR (51.5% total) continue adding Shallow bias (reaching 38.9%), leaving Deep bias virtually flat (10.0%โ12.6%).
- Bias depth tracks pretraining inheritance: SFT-biased prompts with \(\pi_{\text{SFT}} \le 0.1\) match the pretrained top answer in only 20.7% of cases, whereas prompts with \(\pi_{\text{SFT}} > 0.8\) match the pretrained base in 62.3% of cases. Deep biases represent persistent memories from pretraining.
- Deep biases are substantially harder to remove: LoRA-SFT reduces Shallow bias by -10.3 percentage points but Deep bias by only -4.5 points. Similarly, system-prompt optimization via GEPA achieves -5.8 points on Shallow bias but only -1.6 points on Deep bias.
Highlights & Insights¶
- Dual-rate bias depth metric (\(\pi = \text{DR} \cdot \text{FR}\)): Elegantly addresses the vulnerability of single-prompt evaluations by quantifying whether an LLM's preference reflects genuine internal semantic commitment or surface phrasing sensitivity.
- Counter-intuitive alignment dynamics: Demonstrates that post-training alignment (SFT, DPO, RLVR) does not primarily forge deep semantic convictions, but rather introduces pervasive superficial mode collapse tied to specific prompt templates.
- Generalizable distribution flattening: Shows that training on just 60 prompt families with multi-target diversity generalizes across thousands of held-out topics, reducing systemic bias without sacrificing core reasoning benchmarks.
Limitations & Future Work¶
- Diffuse attribution barriers: While specific Deep and Shallow biases can occasionally be traced to isolated training rows in Dolci-Instruct-SFT, the vast majority cannot be attributed to a single example, arising instead from broad, diffuse patterns across millions of documents.
- Restriction to low-stakes domains: To guarantee objective, open-ended diversity evaluation, prompts were drawn from everyday entities (e.g., butterflies, foods, hobbies). Applying bias depth scoring to high-stakes decision settings (e.g., legal interpretation, clinical recommendations, hiring screening) remains a critical next step.
- Reframing LLM dependence: Scenario reframings rely on an external model (GPT-5.6 Luna). While prompts are audited to avoid leading cues, subtle framing styles might still reflect the generator's idiosyncratic phrasing habits.
Related Work & Insights¶
- vs. B-score (Vo et al., 2025) & NoveltyBench (Zhang et al., 2025): Prior benchmarks evaluate response diversity and collapse under a single prompt template. This paper demonstrates that 75% of such detected biases are superficial artifacts that do not survive contextual reframing.
- vs. Artificial Hivemind (Jiang et al., 2026): While Jiang et al. examine cross-model homogeneity under fixed prompts, this work tracks individual bias provenance across the entire pretraining-to-RLVR lifecycle.
- vs. Forcing Diffuse Distributions (Zhang et al., 2024): Expands upon entropy-flattening fine-tuning by evaluating its differential efficacy on Deep versus Shallow biases, demonstrating that weight-level adaptation is indispensable for dismantling pretraining-ingrained biases.
Rating¶
- Novelty: โญโญโญโญโญ Groundbreaking conceptualization of bias depth that decouples surface phrasing artifacts from intrinsic model convictions.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 4,442 grounded prompt families, 133K reframings, 4 target models, and a 4-stage model lineage.
- Writing Quality: โญโญโญโญโญ Exceptionally clear framing, mathematically grounded metrics, and compelling empirical exposition.
- Value: โญโญโญโญโญ Provides an indispensable diagnostic lens and methodology for evaluating and mitigating model collapse across LLM alignment pipelines.