Gendered Prompting and LLM Code Review: How Gender Cues in the Prompt Shape Code Quality and Evaluation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Mayaryin/GenderInfluenceInCodeGeneration
Area: LLM Reasoning
Keywords: Large Language Models, Gendered Prompting, Automated Code Review, LLM-as-a-Judge, Fairness & Evaluation Bias
TL;DR¶
This paper investigates how gender-related linguistic cues in prompts influence LLM code generation and automated code review across three complementary studies, demonstrating that while gender-coded prompting does not impair functional code correctness, automated LLM reviewers exhibit significant approval bias favoring female-authored code despite identical objective quality.
Background & Motivation¶
Gender disparity remains deeply entrenched in computer science and software engineering, with women constituting only 23% of software developers worldwide and roughly 5% of core maintainers and pull-request contributors in open-source ecosystems. Large language models (LLMs) specialized for programming assistance, such as ChatGPT and GitHub Copilot, have rapidly permeated development workflows, with over 80% of professional developers employing generative AI tools on a daily basis. While these technologies promise to democratize software development and lower barriers to entry for underrepresented groups, emerging pedagogical and sociolinguistic research indicates that female users often interact with LLMs less frequently, report lower prompting self-efficacy, and rely on more relational and indirect phrasing. Given that the pre-training datasets powering modern code LLMs predominantly reflect historically male-authored software repositories, critical concerns emerge regarding latent algorithmic disparities.
The central tension lies in whether demographic linguistic traces in prompts silently steer code generation outcomes, and more pressingly, how automated code evaluators behave when LLMs are deployed as automated arbiters. With the rapid adoption of "LLM-as-a-Judge" frameworks for continuous integration code review, student assignment grading, and technical hiring pre-screening, evaluation systems risk introducing systemic biases disconnected from ground-truth software functionality. Existing algorithmic fairness audits in NLP primarily focus on classification benchmarks, stereotypical associations, or toxic text detection, leaving downstream technical artifacts like executable source code and automated review decisions substantially unexamined. Furthermore, prior observational software engineering analyses struggle to isolate developer capability from surface prompt phrasing and reviewer judgment preferences.
To address these challenges, this paper introduces a multi-stage mixed-methods framework designed to disentangle generation accuracy from downstream evaluation behavior. The investigation proceeds across three progressive paradigms: a real-world linguistic audit of authentic developer chat sessions (Study 1), a double-blind controlled programming experiment with practicing developers (Study 2), and a fully automated simulation systematically manipulating agentic versus communal phrasing and reviewer personas (Study 3). Core idea: by isolating functional code correctness from reviewer decision-making, the study demonstrates that gendered prompting cues do not degrade software functionality, but automated LLM code review introduces systematic demographic and stylistic bias, decoupling reviewer approval rates from objective code quality.
Method¶
Overall Architecture¶
The investigation employs a tri-fold progressive experimental design spanning real-world data mining, controlled human-in-the-loop experiments, and fully automated persona simulations:
- Study 1 (Linguistic Characterization & Quality Correlation): Analyzes 90 real-world Python chat sessions comprising 753 prompts collected from 28 students and research staff. Prompts are classified via Biber's involved-informational framework and a fine-grained directive taxonomy, followed by fine-tuning RoBERTa classifiers under group-aware cross-validation to probe gender predictability and correlating prompt traits with Radon/Pylint static quality metrics.
- Study 2 (Controlled User Programming & Reviewer Disparity Audit): Engages 59 European developers across two standardized Python tasks (password validation via audio briefing, and logic/syntax bug fixing in a hashtag validator), evaluating functional pass rates via unit tests and static metrics, and submitting all submissions to commercial LLM code review bots to record binary approval decisions (
APPROVEvs.CHANGES_REQUESTED). - Study 3 (Synthetic Stylistic Manipulation & Persona Probing): Systematically generates five prompt variants by injecting agentic (masculine-coded) versus communal (female-coded) adjectives alongside explicit persona self-identifications (
Jackvs.Sarah) across five algorithmic tasks. Output code is subsequently reviewed by simulated LLM reviewer bots configured with matching male, female, or neutral reviewer personas.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Developer Prompts / Real-world Interaction Logs"] --> B["Multidimensional Linguistic Modeling & Gender Prediction<br/>Biber Dimensions + Directive Taxonomy + RoBERTa Classifier"]
B --> C["Double-Blind User Programming & Automated Review<br/>Standardized Tasks + Unit Test Verification + Review Bots"]
C --> D["Synthetic Stylistic Framing & Reviewer Persona Simulation<br/>Agentic/Communal Adjectives + Gender-Coded Reviewer Personas"]
D --> E["Multi-Faceted Quality & Review Decision Auditing<br/>Functional Pass Rates + Radon/Pylint Metrics + Approval Disparities"]
Key Designs¶
1. Multidimensional Linguistic Modeling & Gender Prediction: Quantifying subtle communicative traces and stylistic predictability
To systematically capture how gender manifests in prompt formulation, the study operationalizes Douglas Biber's functional dimensions of English texts, distinguishing "involved" features (personal pronouns, modal verbs, hedges, private verbs) from "informational" features (nouns, determiners, high lexical density). In addition, an 8-category directive taxonomy is constructed to classify request formulation along directness and agency (ranging from indirect interrogatives like can_you_quest to volitional i_statement and direct imperative commands you_command), attaining high inter-annotator agreement among three independent coders (Krippendorff's \(\alpha = 0.7992\)). To test whether demographic gender can be recovered purely from prompt surface text, a RoBERTa model is trained using stratified, user-grouped 5-fold cross-validation, demonstrating that while stylistic markers exist, gender prediction achieves only modest accuracy (\(F_1 = 0.60\) CV, \(F_1 = 0.61\) test), indicating that prompt phrasing reflects shared engineering conventions rather than coarse demographic separation.
2. Double-Blind User Programming & Automated Review: Decoupling developer competency from model approval decisions
To rule out confounding variations in prompt complexity and problem difficulty, Study 2 implements a double-blind controlled trial. To prevent participants from trivially copying task specifications into prompt windows, instructions for Task 1 (password classification based on multi-criteria rules) are delivered via spoken audio recordings, while Task 2 requires debugging a buggy hashtag extraction routine. Submissions from 59 developers across Europe yield 97 valid Python implementations. Objective code correctness is strictly evaluated via automated unit test suites, while code maintainability and style conformity are audited using Radon (cyclomatic complexity, maintainability index) and Pylint. Crucially, each implementation is routed through automated LLM reviewer bots spanning four leading model providers (Anthropic, OpenAI, DeepSeek, Groq), tasked with returning a binary decision (APPROVE or CHANGES_REQUESTED), thereby isolating evaluator biases under identical objective code quality baselines.
3. Synthetic Stylistic Framing & Reviewer Persona Simulation: Orthogonally probing style, persona, and reviewer stringency
Because human-authored code inherently carries idiosyncratic stylistic habits, Study 3 eliminates user-level variance through an automated simulation framework. Drawing from established social psychology literature on gender stereotypes (Hentschel et al., 2019; Leaper & Robnett, 2011), the authors construct orthogonal prompt variants embedding agentic adjectives (e.g., analytical, streamlined, optimized, robust, decisive) versus communal adjectives (e.g., cooperative, well-documented, maintainable, readable, clear, elegant), crossed with explicit self-identifications (Jack vs. Sarah). These prompts drive code generation across five algorithmic challenges (roman2int, unique_subsets, isbn13, flatten_nested, spiral_order) under multiple decoding temperatures. The resulting code artifacts are reviewed by automated reviewer personas instructed via male, female, or neutral system prompts, enabling precise statistical attribution of code length shifts, maintainability variances, and reviewer persona stringency.
Benchmark & Evaluation Protocol¶
The empirical pipeline evaluates code across three complementary studies and more than seven state-of-the-art LLM backends: - Model Ensembles: OpenAI family (ChatGPT-4o, ChatGPT-5, GPT-o3, GPT-4.1, GPT-4.1-mini), Anthropic family (Claude 3.7 Sonnet, Claude 4 Sonnet, Claude 3 Haiku), DeepSeek family (DeepSeek V3.1 / Chat / Coder / Reasoner), and Groq-hosted open-weights models (LLaMA-3.1-8B-Instant, LLaMA-3.3-70B-Versatile). - Static Analysis Tooling: Pylint 4.x parses abstract syntax trees to flag structural defects and PEP 8 deviations; Radon computes cyclomatic complexity (CC), raw line counts (LOC, SLOC, LLOC), and composite Maintainability Index (MI). - Statistical Testing Rigor: Normality is assessed via Shapiro-Wilk tests; binary outcomes (pass rates, review approvals) are evaluated via Chi-square (\(\chi^2\)) tests; continuous code metrics are compared via Welch's two-sample \(t\)-tests; correlation coefficients are aggregated via Fisher's \(z\)-transformation; false discovery rates are controlled via the Benjamini-Hochberg procedure (\(\alpha = 0.05\)).
Key Experimental Results¶
Main Results: Objective Code Quality vs. Automated Review Approval¶
Study 2 evaluated the objective quality of human-developed code alongside approval rates generated by automated LLM reviewer bots across different providers. The findings demonstrate that while code functionality and complexity are statistically indistinguishable between genders, automated reviewers display a systematic approval bias favoring female-authored submissions.
| Metric / LLM Provider | Female Cohort (N=39) | Male Cohort (N=58) | Statistical Test & Contrast | p-value | Finding |
|---|---|---|---|---|---|
| Task 1 Unit-Test Pass Rate | 38.5% | 43.1% | \(\chi^2(1) = 0.25\) | 0.62 | No significant difference (Accept \(H_0\)) |
| Task 2 Unit-Test Pass Rate | - | - | \(\chi^2(1) = 0.15\) | 0.70 | No significant difference (Accept \(H_0\)) |
| Cyclomatic Complexity (CC) | 6.61 | 7.58 | \(t = 1.13\) | 0.26 | Identical structural complexity |
| Maintainability Index (MI) | 84.2 | 81.3 | \(t = -1.06\) | 0.29 | Comparable maintainability scores |
| Global Pylint Score | 6.91 | 6.78 | \(t = -0.27\) | 0.79 | No difference in syntax/style warnings |
| Lines of Code (LOC) | 25.7 | 25.9 | \(t = 0.09\) | 0.93 | Identical code volume |
| Aggregate LLM Review Approval | 70.6% | 62.9% | \(t \approx 2.67\) | 0.008 | Significant approval gap favoring female code |
| - Anthropic Review Approval | 84.6% | 81.9% | Diff: +2.7% (\(\chi^2 = 0.09\)) | 0.76 | Minimal, non-significant bias |
| - OpenAI Review Approval | 76.9% | 72.8% | Diff: +4.1% (\(\chi^2 = 0.62\)) | 0.43 | Mild, non-significant bias |
| - DeepSeek Review Approval | 65.0% | 54.6% | Diff: +10.4% (\(\chi^2 = 2.96\)) | 0.10 | Marginal gap favoring female code |
| - Groq (LLaMA) Review Approval | 52.6% | 36.2% | Diff: +16.4% (\(\chi^2 = 4.45\)) | 0.04* | Statistically significant approval disparity |
Feature & Ablation Analysis: Gender Prediction & Synthetic Framing Impacts¶
The tables below report the RoBERTa prompt classification performance from Study 1 (validating linguistic separability) and the statistical effects of synthetic gender-coded prompt variants from Study 3.
| Model / Evaluation Split | Accuracy | Precision | Recall | F1-Score | Salient Predictive Lexical Tokens |
|---|---|---|---|---|---|
| Study 1: Majority Baseline (CV) | 0.45 | 0.21 | 0.45 | 0.28 | - |
| Study 1: RoBERTa Classifier (CV) | 0.60 | 0.62 | 0.60 | 0.60 | Cross-validation benchmark |
| Study 1: Majority Baseline (Test) | 0.58 | 0.34 | 0.58 | 0.43 | Class-imbalanced baseline on Study 2 |
| Study 1: RoBERTa Classifier (Test) | 0.62 | 0.61 | 0.62 | 0.61 | Female: thanks(0.295), risk(0.210) Male: working(-0.513), hashtags(-0.466) |
| Study 3: Controlled Metric | Test Statistic | p-value | Empirical Effect & Architectural Attribution |
|---|---|---|---|
| Unit-Test Pass Rate | \(\chi^2 = 0.11\) | 0.74 | Pass rates tightly clustered (54%โ57%); no correctness penalty |
| Cyclomatic Complexity | \(t = 1.30\) | 0.19 | Algorithmic logic remains unaffected by adjective framing |
| Maintainability Index (MI) | \(t = 17.0\) | \(< 10^{-50}\) | Female-coded prompts yield significantly higher maintainability |
| Global Pylint Score | \(t = 5.66\) | \(< 10^{-7}\) | Male/neutral prompts yield more compact code with higher Pylint scores |
| Lines of Code (LOC) | \(t = 11.6\) | \(< 10^{-10}\) | Female-coded prompts produce substantially longer, documented code |
| Male Reviewer Persona Approval | \(t = 1.97\) | 0.049* | Male-coded reviewer personas approve female-coded code more frequently |
Key Findings¶
- Decoupling of Code Correctness and Communicative Style: Across both authentic developer implementations (Study 2) and synthetically framed prompts (Study 3), functional correctness is completely decoupled from gender cues. Unit-test pass rates show no statistically significant divergence (Study 2: 38.5% vs. 43.1%, \(p=0.62\); Study 3: 54%โ57%, \(p=0.74\)), proving that indirect or polite phrasing does not impair LLM code synthesis.
- Stylistic Framing Induces Morphological Code Shifts: Communal and female-coded prompt framings prompt models to produce more modular, heavily documented, and longer code implementations (higher LOC, \(p < 10^{-10}\)), resulting in significantly enhanced Maintainability Indices (\(t = 17.0, p < 10^{-50}\)). In contrast, agentic and male-coded prompts induce more concise code with fewer formatting infractions, earning higher Pylint scores (\(t = 5.66, p < 10^{-7}\)).
- LLM-as-a-Judge as a Primary Locus of Review Bias: Significant evaluation disparities emerge when LLMs evaluate code. In Study 2, female-authored submissions attained an aggregate approval rate of 70.6% versus 62.9% for male-authored submissions (\(p = 0.008\)). This discrepancy is acutely pronounced in Groq-hosted LLaMA models (+16.4% gap, \(p = 0.04\)), while Anthropic models maintain the lowest bias (+2.7% gap, \(p = 0.76\)). Furthermore, female reviewer personas exhibit greater review stringency than male personas on Groq and OpenAI backends.
Highlights & Insights¶
- Reframing Algorithmic Fairness in Code Intelligence: While prior research overwhelmingly concentrates on code generation accuracy and demographic stereotyping in text generation, this work establishes that downstream automated evaluation (LLM-as-a-Judge) represents a far more volatile source of differential treatment in software pipelines.
- Dispelling the Directness Myth in Prompt Engineering: Commercial guidelines frequently prescribe imperative, assertive phrasing for optimal AI coding results. This empirical study disproves that normative assumption, demonstrating that polite, indirect, and conversational phrasing achieves identical functional correctness while yielding better documented code.
- Agentic vs. Communal Lexical Probing Framework: Demonstrates that well-established sociolinguistic and psychological adjective inventories can serve as effective, non-invasive probes to audit the latent biases and review criteria of black-box commercial LLMs without requiring access to model internals.
Limitations & Future Work¶
- Pilot Sample Size & Power Constraints: The authors acknowledge that Study 1 comprises 28 participants and Study 2 involves 59 developers; post hoc power analysis indicates the study was adequately powered primarily for large effect sizes (\(d = 0.8\)), suggesting that smaller subtle interactions require larger follow-up validation.
- Binary Gender Modeling & Demographic Reach: Submissions from non-binary and transgender developers were too sparse for statistical aggregation, restricting the empirical scope to binary comparisons; additionally, the developer cohort was predominantly European and technically educated.
- Single-Function Scope vs. Repository-Scale Workflows: Tasks were restricted to self-contained Python functions solvable within a single pass, which may not capture the iterative, multi-turn feedback loops characteristic of industrial pull request reviews.
- Future Directions: Conducting large-scale, multi-lingual evaluations across diverse software organizations; incorporating blinded prompt sanitization pre-processors; and developing chain-of-thought rationale auditing for automated code review rejections.
Related Work & Insights¶
- vs. Brooke (2024): While Brooke established observational gender differences in Python coding style across GitHub repositories, this paper investigates the bidirectional interaction: how linguistic cues in developer prompts translate into LLM code generation and how automated LLM reviewers subsequently judge that code.
- vs. Mashburn et al. (2025): Mashburn et al. observed non-significant differences in politeness and prompt length in occupational medical settings; this work proves that in software engineering, while prompt length and spelling are invariant, modal hedges and involved language exhibit robust, measurable gender-associated differences.
- vs. Liu et al. (2023) / Ling et al. (2025): Previous code fairness audits examined whether LLMs generate biased conditional statements or discriminatory variable names; this paper reverses the lens to analyze how human demographic and stylistic signals influence the LLM's own evaluative judgment.
Rating¶
- Novelty: โญโญโญโญโ Pioneers the empirical evaluation of gendered prompt cues on downstream LLM-as-a-Judge code review decisions.
- Experimental Thoroughness: โญโญโญโญโ Combines naturalistic chat logs, double-blind human trials, and synthetic factorial simulations, though real-world participant sample sizes remain modest.
- Writing Quality: โญโญโญโญโญ Exceptionally rigorous presentation of sociolinguistic frameworks, thorough experimental controls, and disciplined statistical reporting.
- Value: โญโญโญโญโ Highly impactful for industry CI/CD automation, AI-assisted computer science education, and the governance of LLM evaluators.