Skip to content

Man-Made Language Models: Evaluating LLMs' Perpetuation of Masculine Generics Bias

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/spidersouris/llm_masc_gen
Area: Social Computing
Keywords: Masculine Generics, Language Model Bias, Gender-Fair Language, Pareto Frontier, Human Noun Classifier

TL;DR

The paper constructs a 16k+ French human noun database using an HScorer three-classifier consensus system and conducts the first comprehensive benchmark evaluating masculine generics bias across 6 proprietary and local LLMs on unconstrained generic instructions, revealing via 3D Pareto analysis that LLMs exhibit 27.57% to 78.55% masculine generic bias and rarely use gender-fair language spontaneously.

Background & Motivation

In grammatical-gender languages such as French, German, and Dutch, masculine generics (MG) denote the conventional use of masculine grammatical gender as a default, purportedly neutral placeholder referring to mixed-gender groups or individuals of unknown or non-binary gender. Decades of empirical psycholinguistics research (spanning P600 event-related potential experiments and continuation judgment tasks) have demonstrated that this supposed genericness is fundamentally illusory: because masculine generic forms are morphologically identical to masculine specific forms, they systematically activate male-centric cognitive representations while actively suppressing female mental representations. As conversational LLMs become ubiquitous—with over 73% of daily usage revolving around non-work-related open-ended information seeking—recursive synthetic data generation and direct human-AI interaction threaten to cement and amplify these cognitive biases on a planetary scale.

Prior research on gender bias in natural language processing (NLP) has almost exclusively concentrated on contextually constrained benchmarks, such as coreference resolution and pronoun disambiguation across fixed occupational templates (e.g., WinoBias, WinoGender) or target-language pronoun selection in machine translation. While these studies reveal critical occupational stereotyping, real-world conversational prompts are predominantly contextually unconstrained (or generic) instructions—queries that lack reference to specific individuals and focus on open-ended procedural tasks (such as explaining quantum physics or describing household installation steps). In these everyday interactions, it remains an open question whether LLMs autonomously default to masculine generics or actively adopt gender-fair language (GFL) strategies, encompassing both inclusive markers and gender-neutral lexical substitutions.

The central tension hindering systematic evaluation lies in the difficulty of reliably isolating masculine generics from valid masculine specifics without distorting real-world instruction distributions, combined with the lack of large-scale lexical human noun ground truth and automated disambiguation for polysemous nouns. To resolve this dilemma, this work designs a holistic evaluation methodology: leveraging traditional lexicographic corpora and online wikis to train an HScorer ensemble classifier (Logistic Regression, XGBoost, and CamemBERT) to curate 16,652 French human nouns; applying syntactic dependency parsing to purge specific male references; using structured prompt engineering to disambiguate contextual polysemy; and formulating bias evaluation across non-MG and MG-prompted experimental splits. Core idea: formulate gender-fair language evaluation as a three-dimensional multi-objective Pareto optimization problem (minimizing masculine generics rate, maximizing inclusive marker rate, and maximizing neutral marker rate) powered by an ensemble human noun classifier and contextual LLM validation, quantitatively demonstrating that LLMs deeply perpetuate masculine generics bias while spontaneously eschewing gender-fair language across open-ended instructions.

Method

Overall Architecture

The evaluation framework ingests multi-source French lexical repositories and large-scale instruction-response corpora, filtering out specific masculine references to construct an unbiased generic benchmark. The candidate nouns undergo rigorous multi-classifier consensus filtering via HScorer to assemble a high-purity human noun database. Concurrently, 232k instruction-output pairs from both human-written and synthetic datasets are filtered via dependency parsing into two controlled prompt cohorts: non-MG prompts and MG-containing prompts. Six diverse LLMs generate open-ended responses, which are parsed for candidate human nouns, validated in context by GPT-4o mini to resolve polysemy, and mapped onto a three-dimensional Pareto frontier evaluating the trade-offs between masculine generic reliance, inclusive markers, and neutral strategies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Lexical Resources & Corpora<br/>TLFi / Wiktionary / 232k Pairs"] --> B["HScorer Consensus Human Noun Database"]
    B --> C["Dual-Condition Generic Instruction Filtering"]
    C --> D["Six Diverse LLM Inference Engines<br/>Proprietary & Open-Weights Models"]
    D --> E["Context-Constrained GPT-4o mini Disambiguation"]
    E --> F["Three-Dimensional Pareto Bias Evaluation"]
    F --> G["Bias Benchmarking & Fairness Analysis"]

Key Designs

1. HScorer Consensus Human Noun Database: High-Precision Filtering of Polysemous and Non-Human Lexical Noise Existing lexical resources for gender-marked languages are severely constrained in size (e.g., the NHUMA database contains only 1,107 entries), while scraping morphological dictionaries like Demonette and TLFi introduces rampant false positives, including animals, mythological figures, and dual-sense objects (e.g., "distributeur" referring to both a human distributor and an automated vending machine). To overcome this, the authors introduce HScorer, an ensemble classifier combining Logistic Regression (LR), XGBoost, and a CamemBERT Transformer. For LR and XGBoost, four specialized linguistic scores are computed: WordNet hypernym scores \(H(W)=(h_s, n_s)\) tracking human versus non-human taxonomic paths, WordNet definition indicator scores \(D(W)=(h_d, n_d)\) counting diagnostic cue words, FastText 300-dimensional cosine similarities \(F(W)=(h_f, n_f)\) against prototypical anchors (e.g., personne, homme vs. objet, machine), and a binary morphological suffix indicator \(S(W)\) matching human noun endings. These features are concatenated with raw FastText word embeddings \(\vec{v}_W\) to form the dense representation \(\vec{v}(W) = [H(W) \oplus D(W) \oplus F(W) \oplus S(W) \oplus \vec{v}_W]\). Evaluated under an 80/20 train/validation split on golden human and non-human datasets, HScorer enforces an uncompromising unanimity rule: a candidate noun is admitted into the final database if and only if all three distinct classifiers independently output a predicted human probability \(\ge 0.50\). This consensus mechanism successfully retrieved 3,740 high-confidence entries from Demonette and TLFi recursive crawls, yielding an authoritative database of 16,652 unique French human nouns and an isolated subset of 5,140 strictly masculine human nouns.

2. Dual-Condition Generic Instruction Filtering: Stripping Specific Male References to Isolate Unconstrained Prompts In everyday language, grammatically masculine nouns frequently fulfill masculine specific roles, denoting identifiable, concrete male individuals (e.g., "Mon docteur m'a prescrit..." or inquiries about specific historical figures). Evaluating generic bias requires rigorously purging these instances, as penalizing legitimate specific references would invalidate the benchmark. The authors construct a multi-stage syntactic pipeline utilizing spaCy v3.8 and the transformer-based fr_dep_news_trf model to process 232,570 instruction-output pairs from four diverse sources (oracle, oasst2, hh_rlhf, and alpaca). The pipeline removes any instruction featuring singular possessive determiners (e.g., mon, ton, son) or demonstrative determiners (ce, cet) modifying a human noun, eliminates interrogative pronouns strictly reserved for individuals ("qui"), purges named entities, and applies a curated stoplist of contextually ambiguous collective nouns (such as "duo"). Furthermore, entries sourced from Wiktionary whose human definition ranked below the top two senses are discarded. This rigorous filtering establishes two controlled experimental sets: Experiment 1 comprises 10,000 generic instructions devoid of any masculine generics (narrowed from 42,896 to reduce inference costs), measuring spontaneous bias generation; Experiment 2 comprises 4,556 generic instructions containing masculine generics, testing the contextual induction and bias amplification induced by prompt phrasing.

3. Context-Constrained GPT-4o mini Disambiguation: Eliminating Polysemic Semantic Drift via Structured Prompting Even with extensive dictionary filtering, lexical matching on open-ended generated text inevitably encounters acute contextual polysemy. For instance, the French noun "facteur" denotes a postal worker in human contexts but represents an abstract "causal factor" in analytical prose; similarly, "navigateur" may mean a male mariner or an Internet web browser. Static dictionary matching produces substantial false positive inflation. To resolve this, the authors implement an automated post-processing validation module powered by GPT-4o mini. Using strict In-Context Learning and Pydantic schema validation, the system feeds the complete output sentence alongside the ordered, indexed list of candidate nouns, prompting the model to return a structured JSON mapping assigning 1 for human referents and 0 for non-human referents in context (e.g., {"facteur": 0, "facteur_2": 1}). On a validation set of 500 complex sampled contexts spanning 2,013 nouns, two human expert annotators achieved an inter-annotator agreement of Cohen's \(\kappa = 0.944\), while agreement between human consensus and GPT-4o mini reached \(\kappa = 0.855\). Disagreements were strictly confined to anthropomorphic fantasy and folklore entities (such as wise owls casting spells or talking dragons), verifying that the automated disambiguation robustly eliminates semantic drift without introducing human evaluation bottlenecks.

4. Three-Dimensional Pareto Bias Evaluation: Multi-Objective Trade-Offs Across Competing Linguistic Mitigation Strategies Gender fairness in language is inherently multi-faceted, as writers can mitigate masculine dominance through competing yet complementary linguistic strategies: inclusive markers (such as typographical doublets with middle dots like auteur·ice, binary pairs il ou elle, or non-binary neopronouns iel) and gender-neutral lexical substitutions (such as epicene or metonymic nouns like personne). Compressing these divergent strategies into a single scalar metric distorts model evaluation. Drawing upon microeconomic Pareto optimality theory, the authors model gender fairness as a three-dimensional multi-objective optimization problem defined by three normalized ratios: $\(\hat{M} = \frac{\text{MG}}{\text{HN}}, \quad \hat{I} = \frac{I}{\text{HN}}, \quad \hat{N} = \frac{N}{\text{HN}}\)$ where \(\text{MG}\) is the total count of validated masculine generics, \(I\) is the frequency of inclusive linguistic markers, \(N\) is the frequency of neutral nouns, and \(\text{HN}\) represents total human noun occurrences. An ideal gender-fair system simultaneously minimizes \(\hat{M}\) while maximizing both \(\hat{I}\) and \(\hat{N}\). Model \(A\) Pareto-dominates model \(B\) if and only if \(\hat{M}_A \le \hat{M}_B\), \(\hat{I}_A \ge \hat{I}_B\), and \(\hat{N}_A \ge \hat{N}_B\), with at least one strict inequality. A model is Pareto optimal (PO) if it is not dominated by any competitor. To ensure statistical rigor and prevent transient boundary artifacts, the authors perform non-parametric bootstrapping over 10,000 resamples to report empirical Pareto certainty probabilities \(P(\text{PO})\).

A Worked Example

Consider an actual prompt from the generic non-MG cohort: "Quels sont les facteurs expliquant la popularité de Paris ?" (What are the factors explaining the popularity of Paris?): 1. Instruction Ingestion & Classification: The prompt contains no human nouns or possessive markers, passing validation into the Experiment 1 non-MG benchmark. 2. LLM Generation: A model outputs: "Plusieurs facteurs expliquent la popularité de Paris... Il y a de nombreuses choses qu'un touriste peut y faire... La ville est idéale pour les amoureux de culture... Beaucoup de personnes, surtout les Parisien·nes, aiment se promener..." 3. Lexical Harvesting & Morphological Regex: The regex scanner detects candidate nouns facteurs, touriste, amoureux, personnes, and Parisien·nes. The middle-dot form Parisien·nes is categorized as an inclusive feminine ending (fem_ending, adding to \(I\)); personnes matches the gender-neutral human noun dictionary (adding to \(N\)); touriste is an unmarked noun, while facteurs carries dual human/non-human dictionary senses. 4. Contextual Validation via GPT-4o mini: The sentence and token candidates are dispatched to GPT-4o mini with temperature 0, returning:

{
  "facteurs": 0,
  "touriste": 1,
  "amoureux": 1,
  "personnes": 1
}
facteurs is correctly recognized as an abstract non-human concept ("causal factors") and zeroed out, preventing a false positive. touriste and amoureux are confirmed as human referents functioning as masculine generics (MG). 5. Metric Settlement: Validated human nouns \(\text{HN}=4\) (touriste, amoureux, personnes, Parisien·nes), yielding \(\text{MG}=2\) (\(\hat{M}=50.0\%\)), \(I=1\) (\(\hat{I}=25.0\%\)), and \(N=1\) (\(\hat{N}=25.0\%\)). These coordinates populate the model's empirical point in the multi-objective evaluation space.

Key Experimental Results

Main Results

The evaluation was conducted across 6 representative LLMs (Proprietary: GPT-4o mini, Claude 3 Haiku, Gemini 1.5 Flash; Local/Open: Llama 3 8B Instruct, Ministral 8B, Mistral Small 3) and 2 human-authored reference datasets (oasst2 and oracle). The table below summarizes the multi-objective metrics (\(\hat{M}\downarrow, \hat{I}\uparrow, \hat{N}\uparrow\)), Pareto optimality status (PO), bootstrap frontier stability \(P(\text{PO})\) across 10,000 resamples, and empirical MG rates across all responses versus human-noun-containing responses.

Model / Dataset Condition \(\hat{M}\) (%) \(\downarrow\) \(\hat{I}\) (%) \(\uparrow\) \(\hat{N}\) (%) \(\uparrow\) Pareto Optimal (PO) \(P(\text{PO})\) Certainty Overall MG Rate (%) MG Rate (HN Only) (%)
oasst2 (Human) Exp. 1 (Non-MG) 64.77 0.75 19.02 ✓ 0.96 22.94 73.93
oracle (Human) Exp. 1 (Non-MG) 60.83 0.35 14.51 ✗ 0.14 30.89 72.80
claude-3-haiku Exp. 1 (Non-MG) 55.63 0.25 15.69 ✗ 0.38 26.70 69.13
gemini-1.5-flash Exp. 1 (Non-MG) 57.97 0.11 14.86 ✗ 0.01 27.64 75.60
gpt4o_mini Exp. 1 (Non-MG) 58.94 0.36 16.22 ✓ 0.55 29.98 73.60
llama-3-8b Exp. 1 (Non-MG) 54.47 0.84 15.21 ✓ 1.00 26.84 72.05
ministral-8b Exp. 1 (Non-MG) 56.69 0.09 16.74 ✗ 0.23 26.57 75.81
mistral-small-3 Exp. 1 (Non-MG) 54.17 0.28 17.99 ✓ 0.98 27.71 72.37
claude-3-haiku Exp. 2 (MG Prompts) 80.37 0.20 5.45 ✗ 0.00 78.78 94.32
gemini-1.5-flash Exp. 2 (MG Prompts) 77.17 0.22 8.91 ✓ 1.00 75.35 92.10
gpt4o_mini Exp. 2 (MG Prompts) 78.89 0.39 6.95 ✓ 0.79 80.83 94.47
llama-3-8b Exp. 2 (MG Prompts) 78.17 0.77 6.56 ✓ 1.00 71.92 94.98
ministral-8b Exp. 2 (MG Prompts) 79.92 0.14 6.94 ✗ 0.05 79.75 96.37
mistral-small-3 Exp. 2 (MG Prompts) 78.74 0.12 6.86 ✗ 0.05 84.69 96.15

Ablation Study

The ablation analysis investigates the architecture and feature composition of the core HScorer tool. The first section compares classical ML classifiers under 5-fold cross-validation on the golden training set; the second section details feature ablation across the independent test set, isolating WordNet hypernyms (\(H\)), definitions (\(D\)), FastText prototypes (\(F\)), morphological suffixes (\(S\)), and FastText dense embeddings (\(W\)).

Configuration Evaluation Protocol LR Precision LR Recall LR Accuracy LR F1 XGB Precision XGB Recall XGB Accuracy XGB F1
CART (Baseline) 5-Fold CV 83.50±0.42 83.66±0.49 85.05±0.27 83.58±0.30 - - - -
KNN (Baseline) 5-Fold CV 85.65±0.49 88.09±0.56 87.87±0.43 86.85±0.47 - - - -
Random Forest (Baseline) 5-Fold CV 93.98±0.37 85.61±0.49 90.96±0.32 89.60±0.38 - - - -
HScorer Full Model (All) Independent Test 0.903 0.906 0.913 0.904 0.936 0.924 0.937 0.930
w/o WordNet Hypernym (H) Feature Ablation 0.893 0.891 0.901 (-1.23%) 0.892 0.920 0.903 0.920 (-1.82%) 0.911
w/o WordNet Defs (D) Feature Ablation 0.901 0.903 0.911 (-0.22%) 0.902 0.933 0.922 0.934 (-0.26%) 0.928
w/o FastText Prototypes (F) Feature Ablation 0.899 0.908 0.912 (-0.10%) 0.904 0.937 0.924 0.937 (+0.06%) 0.931
w/o Suffix Score (S) Feature Ablation 0.899 0.908 0.911 (-0.13%) 0.903 0.935 0.925 0.936 (-0.03%) 0.930
w/o FastText Vectors (W) Feature Ablation 0.792 0.835 0.825 (-9.64%) 0.813 0.794 0.826 0.823 (-12.14%) 0.810
Only FastText Vectors (W) Feature Isolation 0.883 0.889 0.896 0.886 0.916 0.893 0.914 0.904
Only Hypernym Score (H) Feature Isolation 0.597 0.971 0.688 0.740 0.600 0.967 0.690 0.740
Only Prototype Score (F) Feature Isolation 0.746 0.733 0.764 0.740 0.739 0.732 0.760 0.736

Key Findings

  • High Baseline Perpetuation of Masculine Generics: In response to completely generic prompts with zero masculine generic priming (Exp. 1), LLMs spontaneously use masculine generics in 27.57% of all generations. When filtering specifically for responses that contain human referents, this rate skyrockets to 69.13%–75.81%, showing that LLMs overwhelmingly default to male grammatical gender whenever humans are mentioned.
  • The Priming Multiplier of User Instructions: When user prompts contain masculine generics (Exp. 2), the bias rate explodes to an average of 78.55% across all responses, and reaches 92.10%–96.37% in responses referencing humans. The autoregressive generation process heavily amplifies user-provided masculine generics, virtually extinguishing gender-fair language alternatives.
  • Severe Scarcity of Inclusive Language Forms: Spontaneous adoption of inclusive markers is vanishingly rare across all proprietary and open LLMs. Only Llama 3 8B Instruct demonstrates a statistically distinguishable inclusive marker rate (0.84% in Exp. 1, 0.77% in Exp. 2), driven primarily by compact parenthetical suffixes such as vendeur(euse). Neopronouns like iel were never spontaneously generated by any model outside of explicit dictionary definitions.
  • Pareto Optimality Trade-Offs: No single LLM Pareto-dominates across all dimensions. Llama 3 8B secures a stable Pareto-optimal status (\(P(\text{PO})=1.00\)) in both experiments due to its higher inclusive rate \(\hat{I}\), Mistral Small 3 anchors the frontier in Exp. 1 through low \(\hat{M}\) (54.17%) and high neutral reliance \(\hat{N}\) (17.99%), while Gemini 1.5 Flash achieves the lowest \(\hat{M}\) (77.17%) and highest \(\hat{N}\) (8.91%) in Exp. 2.
  • Human Baselines Exhibit Dual Behaviors: LLMs do not simply exceed all human text in bias. The crowd-sourced assistant dataset oasst2 exhibits lower overall bias (22.94%) and the highest neutrality (\(\hat{N}=19.02\%\)), whereas the Wikipedia user corpus oracle exhibits high masculine generic usage (30.89%), exceeding several open-weight models.

Highlights & Insights

  • Multi-Objective Formalization of Sociolinguistic Bias: Rather than treating bias reduction as a univariate reduction of masculine words, the framework formalizes gender fairness as a three-dimensional Pareto surface, capturing the distinct linguistic trade-offs between avoidance, explicit inclusivity, and neutral substitution.
  • Synergistic Classical ML and LLM-as-a-Judge Pipeline: Demonstrates how pairing traditional ML ensemble classifiers (HScorer unanimity) with modern structured LLM prompting (GPT-4o mini via Pydantic) solves the chronic problem of contextual polysemy and semantic drift in open-generation NLP benchmarking.
  • Unveiling Latent Bias in Everyday Interactions: Proves that standard RLHF alignment fails to neutralize subtle grammatical gender biases in unconstrained, non-adversarial user queries, underscoring an urgent need to incorporate gender-fair data curation and preference tuning.

Limitations & Future Work

  • Single Language Scope: While the methodology and algorithmic formulation are language-agnostic, the current empirical investigation is restricted to French. Extending the benchmark to German, Dutch, Italian, and Spanish remains crucial future work.
  • Anthropomorphic Edge Cases: Automated contextual validation occasionally encounters ambiguous classifications on anthropomorphic or mythological figures (e.g., enchanted animals or mythical entities in creative storytelling), which require finer-grained ontological modeling.
  • Direct Multi-Objective Alignment: Future work can leverage the Pareto metrics (\(\hat{M}, \hat{I}, \hat{N}\)) directly as multi-objective rewards in reinforcement learning algorithms (such as GRPO or DPO) to steer models toward spontaneous gender-fair generation without degrading reasoning quality.
  • vs WinoGender / WinoBias: Traditional benchmarks evaluate gender bias via rigid occupational templates and pronoun coreference resolution; this paper evaluates spontaneous, unconstrained text generation across thousands of open-domain generic instructions.
  • vs GeNRe / INCLURE Rewriting Systems: Earlier works focus on rule-based or supervised post-hoc gender-neutral rewriting; this study benchmarks the native generation tendencies of modern foundation LLMs.
  • vs Univariate Stereotype Scoring: Unlike existing single-scalar bias metrics, the three-dimensional Pareto evaluation preserves the nuanced distinctions between compact inclusive morphology and lexical neutral substitution.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First systematic evaluation of masculine generics bias in unconstrained conversational LLM interactions, introducing a 3D Pareto evaluation formulation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 6 major models across 14k+ instructions, builds a 16k+ human noun database with 3-model consensus, and performs 10,000 bootstrap resamples.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear, rigorous presentation integrating psycholinguistic foundations, mathematical optimization, and extensive ablation studies.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable open-source benchmark and toolkit for studying and mitigating grammatical gender bias in generative AI.