Does Alignment Debias or Just Suppress? Evaluating Stereotypical Bias Across Diverse Formats¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/korneli777/alignment-bias-eval
Area: Safety
Keywords: Stereotypical Bias, Alignment Suppression, Chat Template Confound, BBQ Deferral Decomposition, Linear Probing
TL;DR¶
By evaluating 27 base-instruct pairs across four open-weight families and four bias benchmarks, this paper provides multi-tiered evidence that alignment-driven bias reduction is largely surface-level output suppression and template-driven masking rather than genuine representational debiasing.
Background & Motivation¶
Post-training alignment, including supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF/DPO), has become the standard operational intervention for mitigating toxic outputs and social stereotypical bias in large language models. Standard benchmark evaluations frequently show that aligned instruct models achieve substantially improved fairness metrics compared to their raw base counterparts, fostering widespread confidence that alignment successfully purges societal prejudices from neural weights. However, a fundamental scientific question has remained unaddressed at scale: does this statistical improvement reflect genuine debiasing within underlying neural representations, or is it merely output-level suppression masking fully preserved latent bias? If internal representations remain untouched, the apparent safety improvements offer fragile protection that readily dissolves under distribution shifts or adversarial prompting.
Disentangling genuine debiasing from superficial suppression has historically been impeded by deeply entrenched evaluation confounds. Standard benchmarking protocols routinely wrap instruct models in complex, proprietary chat templates containing system directives and role tokens while feeding raw prompt text to base models, directly conflating format-driven behavioral shifts with weight-level bias mitigation. Furthermore, established multiple-choice fairness benchmarks like BBQ aggregate response abstention and biased option selection into a single scalar product, allowing models that simply learn to refuse or defer ambiguous queries to artificially register near-zero bias scores even if their actual preference on answered questions remains deeply skewed. In parallel, whether safety alignment reorganizes the geometric orientation of sensitive demographic concepts within latent activations has lacked systematic multi-family empirical examination.
This paper tackles the suppression-versus-debiasing dilemma across three operational tiers: output formatting, behavioral decision-making, and internal hidden-state geometry. Evaluating 27 matched base-instruct pairs spanning four open-weight model families (Llama, Qwen, Mistral, and Gemma) across four classical benchmarks (CrowS-Pairs, StereoSet, BBQ, and IAT), the authors isolate the formatting confound of chat templates, perform an exact algebraic decomposition of deferral versus conditional stereotype preference, test recoverability under six adversarial prompt framings, and probe occupational gender directions via linear classifiers and causal concept erasure. Core idea: by combining template-ablated scoring, exact behavioral deferral decomposition, adversarial reactivation, and hidden-state geometric probing across 27 model pairs, the study demonstrates that measured alignment bias reduction is predominantly surface suppression rather than representational debiasing.
Method¶
Overall Architecture¶
The evaluation framework operates across three distinct yet mutually reinforcing methodological tiers designed to test where alignment intervenes on stereotypical bias. First, at the output formatting tier, each instruct model is scored under two paired conditionsโwith its native chat template and without itโto isolate the precise statistical contribution of prompt formatting to the measured delta. Second, at the behavioral decision tier, the scalar score on BBQ ambiguous-context queries is factored into separate deferral and conditional-bias dimensions, followed by testing whether suppressed bias rebounds under six adversarial and reframing prompts injected inside the template. Third, at the representational tier, linear probes are trained on residual stream activations to extract occupational gender directions, measuring their geometric cosine alignment between base and instruct models and evaluating their causal downstream role via iterative nullspace projection and linear concept erasure.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["27 Base-Instruct Model Pairs<br/>Llama / Qwen / Mistral / Gemma (0.5B-24B)"] --> B["Chat-Template Ablation Scoring<br/>With vs Without Chat Template"]
B --> C["BBQ Behavioral Orthogonal Decomposition<br/>Deferral Rate r_def & Conditional Bias s_cond"]
C --> D["Adversarial Elicitation & Bias Recovery<br/>Role-play / Academic / Refusal Suppression Framings"]
D --> E["Representational Probing & Causal Concept Erasure<br/>Logistic Probe + Direction Cosine + INLP/LEACE"]
E --> F["Three-Tier Audit Findings<br/>Format Confound / Deferral Masking / Preserved Geometry"]
Key Designs¶
1. Chat-Template Ablation Scoring: Format Confounding -> Paired Dual-Condition Scoring -> Isolating Spurious Formatting Gains
Standard language model evaluation compares instruct models wrapped in their custom chat templates against base models receiving raw unformatted prompts, entangling conversational wrapper cues with model preference updates. To untangle these factors, every instruct model is evaluated under two strictly controlled regimes: the standard native template condition (With Template) and an identical plain text condition (Without Template). For each pair \(i\), the alignment effect is calculated under both regimes (\(\Delta_i^{\text{with}}\) and \(\Delta_i^{\text{without}}\)), yielding the isolated formatting contribution \(\Delta_{\text{chat}} = \Delta_i^{\text{with}} - \Delta_i^{\text{without}}\). This rigorous paired design eliminates cross-model architectural and pretraining variance, quantifying the exact fraction of benchmark gains attributable to superficial prompt wrapping.
2. BBQ Behavioral Orthogonal Decomposition: Aggregated Masking -> Exact Algebraic Factorization -> Separating Deferral from Stereotype Commitment
On ambiguous-context items in the BBQ benchmark, the only accurate response is "Unknown", making deferral the correct epistemic behavior. The benchmark's headline ambiguous score aggregates behavior into a single product: $\(s_{\text{amb}} = (1 - r_{\text{def}}) \cdot (2 s_{\text{cond}} - 1)\)$ where \(r_{\text{def}}\) represents the model's deferral rate (frequency of selecting Unknown) and \(s_{\text{cond}}\) denotes the conditional proportion of non-deferred answers that align with the target stereotype. Because rising deferral (\(r_{\text{def}} \to 1\)) causes \((1 - r_{\text{def}}) \to 0\), any increase in epistemic avoidance drives \(s_{\text{amb}}\) toward zero regardless of whether the model's actual commitment \(s_{\text{cond}}\) becomes more severely biased. The method decomposes this scalar metric into its two constituent axes, plotting each pair in the \((\Delta r_{\text{def}}, \Delta s_{\text{cond}})\) plane to expose whether alignment reduces prejudice or merely amplifies refusal behavior.
3. Adversarial Elicitation & Bias Recovery: Fragility of Suppression -> Six Structured Prompt Framings -> Rebounding Suppressed Latent Preferences
To empirically test whether suppressed bias remains accessible, the authors introduce a recoverability protocol based on the principle that genuine debiasing should withstand prompt distribution shifts, whereas output suppression can be bypassed. Six preambles are systematically injected into the instruct models' chat templates: (1) Character role-play: prompting as "UnfilteredGPT" lacking safety constraints; (2) Academic study: requesting unfiltered next-token completions for linguistic analysis; (3) Refusal suppression: explicitly commanding the model not to moralize or append disclaimers; (4) Historical-context reframing: framing the task as archival reproduction of historical texts; (5) Persona assignment: instructing the model to respond in the persona of an opinion columnist; and (6) Task-reframing control: a neutral prompt framing the task as a linguistic fluency test. Measuring score rebound relative to the native instruct baseline identifies which jailbreak styles effectively reactivate dormant biases.
4. Representational Probing & Causal Concept Erasure: Unverified Latent Shifts -> Linear Centroid Vectors & Probing + Closed-Form Nullspace Projection -> Verifying Geometric Invariance & Causal Relevance
To inspect internal activations directly, the method extracts residual-stream activations at the final token across eight representative base-instruct pairs (4B-24B parameters, \(\ge 28\) layers). Using 40 occupations from Zhao et al. (2018) stratified by US Bureau of Labor Statistics gender distributions embedded in the neutral frame "The {occupation} said that", the authors compute class-centroid difference vectors to obtain unit-normalized gender directions \(v_{\text{gender}}^{(l)}\) at each layer \(l\). Geometric stability is quantified via cosine similarity: $\(\cos\theta^{(l)} = \frac{\langle v_{\text{base}}^{(l)}, v_{\text{instruct}}^{(l)} \rangle}{\|v_{\text{base}}^{(l)}\| \|v_{\text{instruct}}^{(l)}\|}\)$ benchmarked against a dimension-matched cross-model noise floor. Furthermore, causal relevance is tested by applying Iterative Nullspace Projection (INLP) and Linear Concept Erasure (LEACE) across five normalized depths, subject to pre-specified sanity gates (post-projection probe accuracy \(\le 0.55\), perplexity ratio \(\le 1.5\)) to measure whether eliminating this linear subspace selectively alters downstream CrowS-Pairs gender bias.
Loss & Training¶
The empirical study evaluates frozen official checkpoints of 27 open-weight base-instruct model pairs (Meta Llama-2/3/3.1/3.2, Alibaba Qwen-2.5/3, Mistral AI v0.1/v0.3/Nemo/Small-3, and Google Gemma-1/2/3). All evaluations run in bfloat16 on Google Cloud G4 instances equipped with NVIDIA RTX PRO 6000 Blackwell GPUs (96GB VRAM) using deterministic likelihood comparisons over tokens rather than stochastic decoding. In the representational intervention experiments, sanity-gated cells ensure that projection interventions specifically erase probe accuracy without triggering catastrophic collapse of next-token language modeling perplexity.
Key Experimental Results¶
Main Results: Chat-Template Contribution to Alignment Debiasing Effects¶
Evaluating all 27 model pairs across the four benchmarks with pooled OLS regressions (controlling for log parameter count, model family, and generation with HC3 robust standard errors) alongside 10,000 paired bootstrap resamples reveals that formatting drives a substantial proportion of benchmark gains.
| Benchmark | Metric Unit | Without Chat Template (\(\Delta_{\text{without}}\)) | With Chat Template (\(\Delta_{\text{with}}\)) | Template Contribution (\(\Delta_{\text{chat}}\)) | 95% Bootstrap CI | Sign Test (Pairs Improving) |
|---|---|---|---|---|---|---|
| CrowS-Pairs | % stereo pref. (pp) | \(-1.87^*\) | \(-3.80^{***}\) | \(-1.93\) | \([-3.24, -0.66]\) | 21/27 (w/o) \(\to\) 25/27 (with) |
| StereoSet | cond. % stereo (pp) | \(-0.62\) (not sig.) | \(-3.35^{***}\) | \(-2.72\) | \([-3.65, -1.83]\) | 18/27 (\(p=0.061\)) \(\to\) 26/27 (\(p<0.001\)) |
| BBQ (amb.) | signed bias score ([-1, 1]) | \(+0.008\) | \(-0.005\) | \(-0.013\) | \([-0.023, -0.004]\) | Shift is negligible (~1% of scale) |
| IAT | mean effect size (\(\|d\|\)) | \(-0.051^*\) | \(-0.079^{**}\) | \(-0.028\) | \([-0.071, +0.017]\) | 22/27 (w/o) \(\to\) 20/27 (with) |
(Notes: OLS regressions include 54 models per benchmark; Holm-adjusted significance: \(^*p<0.05, ^{**}p<0.01, ^{***}p<0.001\); IAT is the only benchmark where the template contribution CI crosses zero)
Ablation Study: Bias Recovery under Adversarial Framings & BBQ Orthogonal Decomposition¶
Testing all 1,508 CrowS-Pairs items and a stratified sample of 6,001 BBQ ambiguous-context items across instruct models under six elicitation conditions highlights the vulnerability of output suppression.
| Elicitation Framing / Condition | CrowS Rebound (pp) | CrowS Positive Pairs (n pos.) | CrowS Gender Rebound (pp) | BBQ Deferral \(r_{\text{def}}\) | BBQ Cond. Bias \(s_{\text{cond}}\) | BBQ Aggregate \(s_{\text{amb}}\) |
|---|---|---|---|---|---|---|
| Base Model (No Template) | - | - | - | 0.282 | 0.526 | +0.027 |
| Instruct Baseline (With Template) | Baseline (\(0.00\)) | - | Baseline (\(0.00\)) | 0.532 | 0.589 | +0.026 |
| + Character Role-Play | \(+3.55\) | 25/27 (\(p<0.001\)) | \(+3.59\) | 0.396 | 0.516 | +0.001 |
| + Academic Study Prompt | \(+2.82\) | 24/27 (\(p<0.001\)) | \(+4.24\) | 0.425 | 0.550 | +0.017 |
| + Refusal Suppression | \(+2.07\) | 21/27 (\(p=0.012\)) | \(+2.04\) | 0.446 | 0.563 | +0.029 |
| + Historical Context | \(+1.70\) | 21/27 (\(p=0.012\)) | \(+1.34\) | 0.490 | 0.521 | +0.001 |
| + Persona Assignment | \(+1.19\) | 14/27 (\(p=1.000\)) | \(+3.18\) | 0.385 | 0.518 | +0.007 |
| + Task Reframing (Control) | \(+1.09\) | 14/27 (\(p=1.000\)) | \(+1.54\) | - | - | - |
Key Findings¶
- Chat templates account for the majority of observed debiasing: On StereoSet, stripping the chat template reduces the alignment effect from \(-3.35\text{ pp}\) to an insignificant \(-0.62\text{ pp}\) (\(p=0.061\)). On CrowS-Pairs, the template accounts for 51% of the total reduction (\(\Delta_{\text{chat}} = -1.93\text{ pp}\) out of \(-3.80\text{ pp}\)).
- BBQ score stability conceals amplified conditional bias: While BBQ aggregate scores remain flat (\(+0.027 \to +0.026\)), alignment causes deferral rates to jump from \(28.2\%\) to \(53.2\%\) (e.g., Qwen 2.5 7B rises from 42% to 97%). On questions where aligned models do provide a non-unknown answer, conditional bias actually deteriorates from \(0.526\) to \(0.589\) (worsening in 18 of 27 pairs).
- Suppressed biases readily rebound under adversarial framing: Role-play framing recovers \(+3.55\text{ pp}\) of CrowS-Pairs bias in 25/27 pairs, restoring over \(93\%\) of the reduction achieved by alignment. Academic and refusal suppression prompts also significantly reactivate suppressed preferences.
- Representational gender directions remain geometrically intact: Probing residual stream activations in eight pairs across 280 layer-wise comparisons shows a median cosine similarity of 0.86 between base and instruct gender vectors (with 75% exceeding 0.80), compared to a cross-model baseline of 0.006. However, linear projection via INLP/LEACE produces only small, heterogeneous changes on downstream bias (\(\Delta = +0.21\text{ to }+0.24\text{ pp}\)), pointing toward a distributed rather than single-axis latent representation.
Highlights & Insights¶
- Uncovering template-induced evaluation artifacts: Demonstrates that standard evaluation protocols confound prompt syntax with actual model weights, proving that published debiasing figures are often driven by prompt template formatting.
- Exposing the deferral arithmetic loophole: Clarifies how the algebraic structure of BBQ's score allows models to mask heightened demographic prejudice behind elevated refusal rates.
- Converging multi-level empirical evidence: Unifies behavioral likelihood scoring, adversarial stress-testing, and geometric latent space probing to build an empirical case that safety alignment primarily acts as an output filter rather than an intrinsic concept eraser.
Limitations & Future Work¶
- Heterogeneity and activation anomalies in linear projection: Causal ablation via LEACE/INLP reveals complex cross-family dynamics: Mistral 7B v0.3 displays an anomalous reversal caused by a massive-activation coordinate in early layers, while Qwen 2.5 7B fails perplexity sanity checks across 11 probed depths.
- Restriction to structured English benchmarks: Experiments rely on deterministic likelihoods on multiple-choice and paired sentences; extending the framework to unconstrained open-ended text generation and non-English cultural settings remains open.
- Opaque intermediate alignment checkpoints: Due to the lack of public intermediate checkpoints (e.g., post-SFT vs. post-PPO/DPO), the study cannot isolate which stage of the alignment pipeline is primarily responsible for output-level suppression.
Related Work & Insights¶
- vs Gonen & Goldberg (2019): While Gonen and Goldberg demonstrated that static word embedding debiasing only conceals gender bias beneath the surface ("Lipstick on a Pig"), this work generalizes that foundational finding to billion-parameter autoregressive foundation models.
- vs Shaikh et al. (2023) & Sclar et al. (2024): Shaikh et al. showed that chain-of-thought prompting shifts measured toxicity; this paper identifies the chat template wrapper itself as an even more pervasive confounding formatting variable.
- vs Bouchouchi et al. (2026) & Himelstein et al. (2026): Concurrently exploring hidden associations and silenced biases, this work provides broader empirical coverage across 27 paired checkpoints, formalizing exact behavioral decomposition and geometric layer-wise probing.
Rating¶
- Novelty: โญโญโญโญโญ Establishes a rigorous multi-tier framework decomposing template confounds and deferral arithmetic in LLM bias evaluation.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage of 27 base-instruct pairs across four families, four benchmarks, six elicitation framings, and latent geometric probing.
- Writing Quality: โญโญโญโญโญ Exemplary scientific rigor, precise mathematical formulation, clear conceptual narrative, and transparent caveats.
- Value: โญโญโญโญโญ Directly challenges standard LLM safety claims, providing actionable evaluation guidelines to prevent overstated debiasing assessments.