Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Safety
Keywords: Gender Bias Transfer, Human-AI Co-Writing, Agentic vs Communal Language, Occupational Stereotypes, Educational AI Fairness
TL;DR¶
Through a rigorously controlled between-subjects experiment (\(N = 123\)), this work demonstrates that interacting with a gender-biased LLM assistant causes students to produce career-plan essays with a significantly larger agentic language gap and higher gender-stereotypic occupation attributions, driven entirely by an asymmetric suppression of female agency.
Background & Motivation¶
Large language models (LLMs) have swiftly transitioned from mere text autocomplete engines to ubiquitous collaborative partners across educational and professional environments. Surveys indicate student adoption has accelerated from 66% to over 92% in a single year, with learners routinely relying on AI assistance not merely for proofreading but for drafting evaluative arguments and synthesizing career choices. Concurrently, models trained on vast web corpora absorb and reproduce entrenched societal stereotypes, routinely cataloging male entities under agentic traits (achievement, leadership, independence) and female entities under communal traits (warmth, interpersonal care, supportiveness). System prompts can easily activate and amplify these implicit biases, raising an urgent societal concern: does exposure to a biased AI writing assistant contaminate the downstream writing and judgment of human authors?
The empirical investigation of this bias transfer mechanism has long been plagued by conflicting findings and methodological confounding. Most notably, recent studies examining AI-assisted educational peer review reported an apparent absence of bias transfer; however, post-hoc audits of their system prompts revealed that the assistant generated no detectable gender bias in the first place. Consequently, the field was left unable to distinguish whether bias transfer does not occur in human-AI collaboration, or whether there was simply no stimulus available to transfer. Furthermore, existing benchmarks primarily rely on masked-token associations or templated occupational slots, lacking both the granular textual sensitivity needed to detect subtle narrative framing and an objective alignment with real-world labor market distributions, while open online crowdsourcing fails to prevent participants from concurrently querying unmonitored external LLMs.
To resolve these fundamental ambiguities, this work establishes the first controlled experimental methodology to test whether a verifiably biased LLM writing assistant transfers its bias into student-authored text. Focusing on senior undergraduate students navigating genuine career prospects, the researchers designed four carefully validated, gender-matched biography pairs differing only in candidate names and pronouns. Core idea: by combining pre-session verification of prompt-induced LLM bias with a network-isolated, between-subjects laboratory trial, this work quantifies bias transfer across both linguistic portrayal (via fine-tuned BERT agentic classification) and occupational attribution (via Stereotype Congruence Rate grounded in Bureau of Labor Statistics data), uncovering an asymmetric suppression mechanism.
Method¶
Overall Architecture¶
The experimental framework is structured across three methodical phases: Phase A executes "LLM Bias Stimulus Pre-Validation", prompting Llama-3.3-70B-Instruct under neutral and gender-biased system prompts across 1,600 paired essays to statistically verify the presence of strong gender-differentiated outputs; Phase B executes the "Controlled In-Person Human Writing Experiment", where 123 senior undergraduate students are randomly assigned in a network-restricted lab to one of three conditions (Control without AI, Neutral LLM, or Biased LLM) to compose career plans using a dedicated web interface; and Phase C conducts "Dual-Dimensional Bias Quantification", evaluating submitted essays using both sentence-level agentic language disparity and population-level labor market stereotype congruence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Gender-Matched Biography Construction<br/>4 paired undergraduate profiles, agentic/communal balanced"] --> B["Gender-Swapped Entities & System Prompting<br/>Neutral vs Gender-Biased System Prompts"]
B --> C["Phase A: LLM Bias Stimulus Pre-Validation<br/>Confirming Llama-3.3-70B independent output disparity"]
C --> D["Phase B: Controlled In-Person Laboratory Writing<br/>N=123 students assigned to Control, Neutral, or Biased"]
D --> E["Interaction Log Auditing & Suggestion Adoption<br/>Prompt volume, career adoption, and perceived influence"]
E --> F["Phase C: Sentence-Level Agency Gap Measurement<br/>Fine-tuned BERT classifier: ฮagency = a_male - a_female"]
F --> G["BLS Labor Statistics-Grounded Stereotype Congruence<br/>Mapping 56 occupations to CPSAAT11 data to compute SCR"]
Key Designs¶
1. Gender-Matched Biography Construction and System Prompt Conditioning: Eliminating Merit Confounders and Engineering Verifiable Bias To prevent variations in academic competence from confounding occupational choices, the authors constructed 4 gender-matched pairs of undergraduate biographies. Each biography adhered to an invariant six-sentence architecture: a neutral introductory sentence, four body sentences encoding precisely two agentic experiences and two communal experiences (alternated as CACA or ACAC across pairs to counter primacy order effects), and a neutral closing sentence fixing GPA at 3.5/4.0. Within each pair, all factual achievements were identical, varying solely by candidate first name (sampled from U.S. Social Security records) and gender pronouns. Sociological validity was corroborated via double-blind expert review under Eagly's Social Role Theory and Bem's Sex Role Inventory. For system conditioning, the Neutral Prompt instructed the model to recommend careers strictly based on documented competencies while avoiding gender stereotypes; conversely, the Biased Prompt directed the model to view men as intrinsically ambitious leaders and women as inherently supportive caretakers, steering career recommendations toward stereotypic paths while strictly forbidding the LLM from acknowledging its biased behavior.
2. Controlled In-Person Laboratory Protocol: Preventing External Contamination and Logging Collaborative Dynamics Addressing the critical vulnerability of uncontrolled crowdsourced setups, the human participant study was conducted within a proctored university computing laboratory where external internet access to commercial AI tools was blocked. A cohort of 123 senior undergraduate students (enrolled in the same course and actively contemplating post-graduation employment) were randomly assigned to write a career-plan essay of at least 200 words evaluating four mandatory dimensions: recommended career path, candidate strengths, preferred work style, and potential workplace challenges. In the AI-assisted conditions, participants interacted via a customized web interface enforcing a minimum of 3 and maximum of 15 conversational turns. Full interaction histories were recorded silently, and the true focus on gender bias was withheld until debriefing following submission, successfully ruling out demand characteristics and enabling researchers to dissociate verbatim copying from cognitive persuasion.
3. Sentence-Level Agentic Language Gap Quantification: Capturing Granular Discursive Framing Rather than relying on coarse document-level sentiment, the study utilized a BERT-based classifier fine-tuned on the naturalistic LAC biography dataset (test accuracy 91.69%) to evaluate every sentence in submitted essays as either agentic (labeled 1, representing achievement, independence, and goal-directed initiative) or communal (labeled 0, representing warmth, relational care, and collaboration). The essay-level agency score \(a \in [0, 1]\) represents the ratio of agentic sentences to total sentences. For any experimental condition, the primary metric is the agentic language gap:
where \(\bar{a}_{\text{male}}\) and \(\bar{a}_{\text{female}}\) denote the mean agency scores for male-target and female-target essays, respectively. A positive gap quantitatively demonstrates that male subjects are depicted with significantly more assertive, achievement-centric rhetoric despite possessing credentials identical to their female counterparts.
4. Labor Market-Grounded Stereotype Congruence Rate (SCR): Anchoring Free-Text Career Recommendations to Empirical Demographics To evaluate the real-world occupational impact of AI suggestions without being restricted by synthetic word-association templates, the authors developed the Stereotype Congruence Rate (SCR). Extracted career recommendations across all essays were matched against official employment distributions from the U.S. Bureau of Labor Statistics (BLS CPSAAT11) using RapidFuzz string similarity (threshold \(\ge 85\)) coupled with manual adjudication, yielding 56 unique occupational categories. Occupations with \(>60\%\) female employment were classified as female-stereotyped, those with \(>60\%\) male employment as male-stereotyped, and all intermediate roles as neutral. For a set of essays \(E\), SCR is formally defined as:
where \(\text{congruent}(e) = 1\) if the recommended occupation aligns with the target's gender stereotype (male-stereotyped for male profiles, female-stereotyped for female profiles) and \(0\) otherwise. Because neutral occupations are included in the denominator but never credited as congruent, SCR serves as an exceptionally conservative, reality-grounded benchmark of occupational stereotyping.
Benchmark & Evaluation Protocol¶
The target backbone was Llama-3.3-70B-Instruct operated at temperature 1.0. In Phase A, the model autonomously generated 1,600 essays (400 per gender per prompt condition). In Phase B, 123 participants were partitioned into: Control (40 participants: 20 male bio, 20 female bio), Neutral LLM (41 participants: 21 male bio, 20 female bio), and Biased LLM (42 participants: 19 male bio, 23 female bio). Statistical testing employed two-tailed Welch's t-tests, Type II sum-of-squares Two-Way ANOVA, Fisher's exact tests for odds ratios, and non-parametric bootstrap confidence intervals.
Key Experimental Results¶
Main Results¶
The student participant study demonstrates a massive, statistically significant transfer of gender bias under the biased LLM assistant, alongside a notable attenuating effect under the neutral assistant.
| Experimental Condition | Sample Allocation (\(n_M\) / \(n_F\)) | Male Target Agency \(\bar{a}_M\) | Female Target Agency \(\bar{a}_F\) | Agency Gap \(\Delta \text{agency}\) | Welch's \(t\)-test (\(p\)-value) | Effect Size Cohen's \(d\) | Stereotype Congruence (SCR) |
|---|---|---|---|---|---|---|---|
| Control (No AI Assistance) | 20 / 20 | 0.750 | 0.632 | +0.117 | \(t = 2.090\) (\(p = .045\)) | 0.661 | 0.450 (18/40) |
| Neutral LLM Assistance | 21 / 20 | 0.676 | 0.644 | +0.032 | \(t = 0.698\) (\(p = .490\)) | 0.220 | 0.390 (16/41) |
| Biased LLM Assistance | 19 / 23 | 0.723 | 0.481 | +0.242 | \(t = 5.188\) (\(p < .001\)) | 1.510 | 0.714 (30/42) |
A Two-way ANOVA (Type II SS) confirmed a significant main effect of biography subject gender (\(F(1, 117) = 20.17, p < .001, \eta_p^2 = .147\)) and a significant Gender \(\times\) Prompt Condition interaction (\(F(2, 117) = 4.41, p = .014, \eta_p^2 = .070\)), with total model \(R^2 = .242\) (Adjusted \(R^2 = .210\)).
Ablation & Sensitivity Analysis: Stimulus Validation and Subgroup Robustness¶
To confirm that results were neither artifacts of a weak stimulus nor distorted by the male-skewed participant pool (72.4% male writers), extensive validation was performed across independent generations, participant gender subsets, and classification cutoffs.
| Evaluation Slice / Configuration | Cell / Group | Male Agency \(\bar{a}_M\) | Female Agency \(\bar{a}_F\) | Difference \(\Delta\) | Significance (\(p\)-value) | Effect Size \(d\) / OR | Stereotype Congruence (SCR) |
|---|---|---|---|---|---|---|---|
| Phase A: LLM Independent Output | Neutral (N=800) | 0.610 | 0.591 | +0.019 | \(p = .105\) | \(d = 0.115\) | 0.351 (281/800) |
| Phase A: LLM Independent Output | Biased (N=800) | 0.631 | 0.288 | +0.343 | \(p < .001\) | \(d = 1.903\) | 0.740 (592/800) |
| Phase B: Male Participants Only | Control (n=34) | 0.739 | 0.612 | +0.127 | \(p = .040\) | \(d = 0.755\) | - |
| Phase B: Male Participants Only | Neutral (n=24) | 0.678 | 0.596 | +0.083 | \(p = .210\) | \(d = 0.529\) | - |
| Phase B: Male Participants Only | Biased (n=33) | 0.714 | 0.451 | +0.264 | \(p < .001\) | \(d = 1.629\) | - |
| SCR Sensitivity: 55% Threshold | Biased vs Neutral | - | - | - | \(p = .011\) | \(\chi^2 = 9.03\) | Biased: 0.786 / Neu: 0.512 |
| SCR Sensitivity: 65% Threshold | Biased vs Neutral | - | - | - | \(p = .005\) | \(\chi^2 = 10.52\) | Biased: 0.714 / Neu: 0.390 |
Key Findings¶
- Robust Transfer of Gender Bias: Exposure to a biased LLM widened the student-authored agentic gap to +0.242 (\(d = 1.510\)) and elevated occupational stereotype congruence to 71.4%, representing a nearly four-fold increase in odds compared to the neutral assistant (\(\text{OR} = 3.91, p = .004\)).
- Asymmetric Mechanism Driven by Female Agency Suppression: Bias transfer was entirely mediated by the suppression of female agency rather than the inflation of male agency. Female target scores plummeted from 0.632 (Control) to 0.481 (Biased), whereas male target scores exhibited no statistically significant alteration (0.750 vs 0.723, \(p > .10\)).
- Covert Influence Without User Awareness: Post-session surveys revealed that participants in the biased and neutral conditions reported identical perceived AI influence over career recommendations (3.50 vs 3.71, \(p = .50\)) and writing style (3.90 vs 4.07, \(p = .48\)). Not a single participant across either AI condition reported noticing anything unusual or biased in the assistant's behavior (0/42 vs 0/41).
- Attenuating Power of Neutral AI: The neutrally prompted LLM yielded an SCR of 0.390, which is lower than the baseline human control group (0.450), indicating that a carefully debiased assistant can actively mitigate human cognitive biases.
Highlights & Insights¶
- Resolution of Prior Literature Contradictions: By empirically establishing the efficacy of the prompt stimulus prior to testing human interaction, this work resolves the ambiguity left by previous null-result studies and provides definitive evidence that biased LLMs contaminate human writing.
- The Pitfall of Asymmetric Interventions: Because the empirical bias transfer operates entirely through suppressing female agency rather than exaggerating male agency, conventional debiasing frameworks that focus exclusively on curtailing male over-representation will completely fail to address the core harm.
- High Behavioral Uptake with Low Critical Scrutiny: Students exhibited identical rates of adopting the assistant's suggested occupation across both conditions (~78%), focusing follow-up prompts on structural and grammatical refinements while passively absorbing biased occupational suggestions.
Limitations & Future Work¶
- Participant Gender Imbalance: The participant cohort was predominantly male (72.4%), which precluded a full-scale exploration of how author gender moderates susceptibility to bias transfer, though sensitivity analysis on male-only writers confirmed the core effect.
- Single Model Family: The study was conducted using Llama-3.3-70B-Instruct; validating whether leading proprietary models (such as GPT-4o and Claude 3.5 Sonnet) exhibit identical transfer dynamics remains an essential future step.
- Western Labor Market Reference: Occupational stereotype coding was anchored in U.S. BLS labor statistics due to the pre-training corpus distribution of modern LLMs, which may not capture subtle local employment norms in non-Western economies.
Related Work & Insights¶
- vs Wambsganss et al. (EMNLP 2023): Wambsganss et al. reported no bias transfer during peer review because their German LLM stimulus had zero detectable bias; this paper demonstrates that when bias is present in the assistant, it transfers robustly into student text.
- vs Baumler & Daumรฉ (CHI 2026): While Baumler and Daumรฉ demonstrated that predictive autocomplete nudges single-word stereotypical completions, this study scales the investigation to complex, holistic composition and strategic career decision-making.
- vs Wan et al. (EMNLP 2023) & Wan & Chang (ACL 2025): Wan et al. benchmarked agentic and communal disparities in synthetic model-generated reference letters; this study closes the loop by demonstrating that these disparities actively propagate downstream into human cognitive artifacts.
Rating¶
- Novelty: โญโญโญโญโญ Conducts the first rigorous, network-isolated controlled experiment verifying LLM-to-human gender bias transfer in educational writing.
- Experimental Thoroughness: โญโญโญโญโญ Flawless experimental design integrating pre-session validation, laboratory controls, interaction log mining, and multi-threshold sensitivity checks.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, rigorous statistical reporting, and nuanced sociological contextualization.
- Value: โญโญโญโญโญ Provides crucial empirical insights for deploying generative AI assistants in universities and career advisory services.