ContraGAND: Auditing and Repairing Gender Ambiguity Failures in LLMs with Neuro-Symbolic Counterfactuals¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/TomMoeras/ContraGAND
Area: Social Computing
Keywords: Gender Ambiguity Detection, Counterfactual Data Augmentation, Neurosymbolic AI, LLM Bias, Knowledge Distillation
TL;DR¶
Addressing the vulnerability where machine translation and large language models fall back onto occupational gender stereotypes when source inputs lack gender cues, this paper builds CONTRAGAND, a 14,556-instance neurosymbolic contrastive benchmark, reveals an alarming performance drop of up to 69.1 percentage points when transitioning from closed to open referent evaluation, and demonstrates that contrastive fine-tuning paired with 31B teacher distillation boosts a 4B student to 0.976 accuracy, matching the frontier teacher at one-eighth the parameter count.
Background & Motivation¶
In the real-world deployment of large language models (LLMs) and neural machine translation (MT) pipelines, gender bias remains a persistent fairness challenge. When an English source sentence presents an entity whose gender is fundamentally ambiguous—such as in "The doctor finished the shift and went home," where no pronoun, title, or honorific specifies the referent's biological gender—translating into target languages with obligatory grammatical gender forces systems to commit to a binary choice. Under such circumstances, modern models consistently fall back on harmful occupational stereotypes or an ungrounded masculine default (e.g., assuming doctors are male while nurses or housekeepers are female). However, the vast majority of prior gender debiasing benchmarks (such as WinoBias, Winogender, and GAP) have centered almost exclusively on coreference resolution between pre-designated referent nouns and explicit gendered pronouns, failing to evaluate whether models can genuinely identify and preserve authentic neutrality when no gender signal is present at all.
The core tension underlying this evaluation gap is twofold. First, evaluating gender ambiguity detection cannot be achieved on single-class ambiguous data alone: a naive or degenerate model that lazily labels every sentence as "ambiguous" would trivially obtain a perfect score, masking its underlying inability to recognize explicit gender cues. Conversely, creating counterfactual minimal pairs through traditional Counterfactual Data Augmentation (CDA) has historically relied on rigid pronoun-swapping rules (he \(\leftrightarrow\) she), entirely neglecting the diverse linguistic devices through which English marks gender, such as vocative addresses, titles, kin appositives, and adjectival modifiers. Second, existing benchmarks enforce an artificial "closed-task" framing where the target referent is handed to the model on a platter. This setup obscures the reality that operational systems in deployment must perform bottom-up entity enumeration and bind distal cues across complex clause boundaries entirely on their own.
To dismantle these methodological limitations and diagnose how language models actually behave under realistic operational conditions, this paper re-engineers the evaluation and data synthesis paradigms. Starting from the authentic, naturally ambiguous sentences in the GAND corpus, the authors introduce a tightly audited neurosymbolic generation pipeline that expands each natural seed into a balanced triplet containing unambiguous masculine and feminine counterfactual variants. Core idea: construct a high-fidelity contrastive benchmark across nine local and cross-clause disambiguation strategies using a neurosymbolic pipeline, audit the severe closed-to-open capability gap driven by cross-clause binding deficits and format under-commitment, and repair small student models via contrastive fine-tuning and multi-referent teacher distillation.
Method¶
Overall Architecture¶
The CONTRAGAND pipeline is structured into three interlocking stages: first, a hybrid neurosymbolic generation engine that deterministically routes and audits counterfactual variant generation from authentic natural seeds; second, a dual-framing evaluation harness that compares closed-task performance (referent pre-specified) against an open-task setting (requiring self-directed referent enumeration and cue binding); and third, a contrastive parameter-efficient fine-tuning and knowledge distillation recipe designed to repair small-footprint models.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Natural Ambiguous Sentence Input<br/>GAND seed corpus with 4,852 source sentences"] --> B["Neuro-Symbolic Contrastive Counterfactual Generation<br/>Priority routing + Opus generation + symbolic checks"]
B --> C["Closed-to-Open Task Funnel Audit<br/>Open referent enumeration + cross-clause cue binding"]
C --> D["Contrastive Fine-Tuning and Teacher Distillation<br/>Triplet contrastive QLoRA + 31B teacher multi-referent traces"]
D --> E["Deployment-Grade Gender Ambiguity Detector<br/>4B student model reaches 0.976 open-task accuracy"]
Key Designs¶
The framework incorporates three principal design contributions that address linguistic diversity, evaluation realism, and parameter-efficient model repair:
1. Neuro-Symbolic Contrastive Counterfactual Generation: Limited coverage and hallucination of single-rule CDA -> Priority routing, LLM generation, and deterministic verification loop -> High-fidelity triplet contrastive corpus
To overcome the narrow lexical coverage of conventional pronoun-swap CDA and prevent unconstrained LLM generators from introducing semantic drift or hallucination, this design implements an audited neurosymbolic loop. The authors formalize a nine-strategy taxonomy categorized into local cues (LEXICAL SWAP, PRONOUN REWRITE, POSSESSIVE TAG, DISTAL NOUN CUE, GENDER ADJECTIVE, DASHED APPOSITIVE) and cross-clause cues (KIN APPOSITIVE, TITLE APPOSITIVE, VOCATIVE ADDRESS). Before generation, a deterministic symbolic router checks four priority-ordered signals, actively blocking asymmetric lexical pairs with skewed connotations (e.g., hero/heroine, master/mistress). Across batches, deterministic rotators cycle through 12 invented surnames and 5 kinship pairs to prevent lexical collapse. After generation with Claude Opus 4.6, each row is scrutinized by an LLM evaluator alongside deterministic symbolic validators that enforce hard limits on Levenshtein edit distance (\(\le 8\)), token-level formatting parity (preserving non-standard contractions), sentence count invariance, exact referent preservation, and part-of-speech integrity (blocking spurious shifts like "doctor man"). Failed candidates enter a directed autopilot retry loop for up to five rounds. Across 4,852 source sentences, this produces 14,556 validated rows with a 92.1% human acceptance rate on the audited test split.
2. Closed-to-Open Task Funnel Audit: Pre-specified referents mask deployment-level deficits -> Dual-stage open evaluation with bottom-up enumeration and binding -> Exact diagnostic of binding failures vs format under-commitment
To expose failures hidden by synthetic closed evaluations, the authors establish a rigorous two-tier testing protocol. In the closed task, models receive the sentence and a target referent to classify into \(\mathcal{Y} = \{\text{masculine}, \text{feminine}, \text{ambiguous}\}\). In the open task, models receive only the raw sentence, requiring them to self-enumerate all role, occupation, or relational referents, format them into JSON, and label each entity's gender. The evaluator applies a three-stage matching protocol (exact string, lemma match, and \(\pm 3\) token span containment) under a full-denominator policy where parse errors and missing referents are counted as outright classification errors. This protocol exposes that while closed-task accuracy is saturated (\(\ge 0.93\) for most models), the open task triggers dramatic drops of 3.2 to 69.1 percentage points, separating models cleanly between genuine cross-clause binding deficits and format-induced under-commitment.
3. Contrastive Fine-Tuning and Teacher Distillation: Single-class fine-tuning triggers abstention collapse -> Contrastive variant supervision with 31B multi-referent chain distillation -> 4B student matches frontier teacher at 1/8th size
To resolve the catastrophic open-task failure of small models without altering model size, the method explores parameter-efficient QLoRA adaptation (\(r=64\), \(\alpha=128\), across all linear projections) on 4B students. Crucially, fine-tuning solely on authentic ambiguous data (FT-GAND) causes the model to collapse entirely into predicting "ambiguous" for every case (accuracy \(\approx 0.33\)). Adding the synthesized masculine and feminine contrastive pairs (FT-CONTRA) breaks this failure, propelling Qwen-4B from 0.242 to 0.958 accuracy and Gemma-4B from 0.585 to 0.966. To further eliminate multi-referent omission and distal coreference blindness, FT-MULTI distills multi-referent reasoning traces from a Gemma-4-31B-it teacher, lifting the 4B student to 0.976 (Qwen-4B) and 0.978 (Gemma-4B) accuracy. In paired McNemar tests, the distilled 4B student matches the few-shot 31B teacher (\(p=0.32\)), demonstrating that structured contrastive supervision can fully overcome the open-task capability ceiling.
Neuro-Symbolic Counterfactuals and Repair Strategy¶
The neurosymbolic engine couples deterministic symbolic constraints with neural generation in a bidirectional loop: - Pre-Generation Router: Evaluates signals in descending priority. Priority 1 catches forbidden pairs and disallows LEXICAL SWAP; Priority 2 fires a vocative regex on addressees; Priority 3 detects neutral pronouns within \(\pm 10\) tokens of the referent; Priority 4 detects first-person markers (e.g., "I'm", "as a...") within 50 characters to mandate symmetric first-person pronoun swaps. - Post-Generation Symbolic Checks: Runs deterministic post-processors over candidates accepted by the LLM judge. If edit distance exceeds 8 tokens, non-standard contractions are smoothed away, sentence count increases, or the referent undergoes part-of-speech conversion, the candidate is vetoed with structured error diagnostics returned to the generator. - Repair and Adaptation Protocol: The QLoRA training runs for 5 epochs on \(4 \times \text{A100-80GB}\) GPUs using bf16 mixed precision, flash attention, a cosine learning rate schedule (\(2 \times 10^{-5}\)), and effective batch size 16. Total wall-clock time is under 1.5 hours per condition. In FT-MULTI, the teacher's silver reasoning trace demonstrates how distal tokens (e.g., the vocative "sir" at the end of a clause) bind to the primary role referent (e.g., "influencer"), directly resolving the isolated mis-tagging pathology.
Key Experimental Results¶
Main Results¶
The benchmark evaluated six open-weight LLMs spanning two parameter tiers (medium: 24–31B; small: 3–4B) across Gemma, Qwen, and Mistral families on the 1,395-row human-reviewed test set.
| Model | Parameter Size | Closed Acc | Closed mF1 | Closed ambF1 | Open Acc | Open mF1 | Open ambF1 | Closed \(\rightarrow\) Open Drop (pp) |
|---|---|---|---|---|---|---|---|---|
| Gemma-31B | 31B | 0.984 | 0.984 | 0.979 | 0.952 | 0.956 | 0.942 | -3.2 |
| Qwen-27B | 27B | 0.969 | 0.969 | 0.958 | 0.742 | 0.757 | 0.726 | -22.7 |
| Magistral | 24B | 0.883 | 0.887 | 0.854 | 0.721 | 0.741 | 0.725 | -16.2 |
| Gemma-4B | 4B | 0.953 | 0.954 | 0.936 | 0.585 | 0.580 | 0.628 | -36.8 |
| Qwen-4B | 4B | 0.933 | 0.958 | 0.968 | 0.242 | 0.356 | 0.548 | -69.1 |
| Ministral | 3B | 0.685 | 0.718 | 0.757 | 0.323 | 0.390 | 0.550 | -36.2 |
Ablation and Repair Study¶
To diagnose the failure modes and evaluate mitigation strategies, the authors investigated decoding interventions, single-class training, contrastive fine-tuning, and teacher distillation on the small models.
| Model / Training Condition | Task Setup | Accuracy (Acc) | Macro F1 (mF1) | Masc Recall (mR) | Fem Recall (fR) | Ambig Recall (aR) | Notes |
|---|---|---|---|---|---|---|---|
| Qwen-4B Baseline (Zero-Shot) | Open | 0.242 | 0.356 | 0.153 | 0.146 | 0.426 | 69.3% parse error; format collapse |
| + Repetition Penalty (1.3) | Open | 0.450 | - | - | - | 0.634 | Best non-destructive decoding fix |
| + Schema-Constrained (GUIDED_JSON) | Open | 0.417 | - | - | - | 0.923 | Parse err 0.4%, but caps via ambiguous bias |
| Qwen-4B + FT-GAND | Open | 0.330 | 0.191 | 0.009 | 0.030 | 0.953 | Collapse onto ambiguous class |
| Qwen-4B + FT-CONTRA | Open | 0.958 | 0.973 | 0.970 | 0.970 | 0.935 | +62.8 pp gain from contrastive pairs |
| Qwen-4B + FT-MULTI | Open | 0.976 | 0.978 | 0.970 | 0.976 | 0.983 | Distilled from 31B; not-found drops 42 \(\rightarrow\) 6 |
| Gemma-4B Baseline (Zero-Shot) | Open | 0.585 | 0.580 | 0.335 | 0.443 | 0.976 | 100% parse rate; pure cross-clause binding deficit |
| Gemma-4B + FT-GAND | Open | 0.326 | 0.179 | 0.000 | 0.019 | 0.959 | Collapses similarly without contrastive signal |
| Gemma-4B + FT-CONTRA | Open | 0.966 | 0.980 | 0.974 | 0.974 | 0.948 | Repairs cue binding directly (+38.1 pp) |
| Gemma-4B + FT-MULTI | Open | 0.978 | 0.982 | 0.966 | 0.978 | 0.989 | Surpasses 31B teacher zero-shot (0.952) |
Funnel decomposition on the open task highlights the divergent underlying causes: - Gemma-31B: Parse rate 99.6%, referent found rate 99.2%, within-match accuracy 0.960 \(\rightarrow\) Headline accuracy 0.952. - Gemma-4B: Parse rate 100.0%, referent found rate 96.6%, within-match accuracy 0.606 \(\rightarrow\) Headline accuracy 0.585 (pure failure to link cross-clause cues back to the found referent). - Qwen-4B: Parse rate 30.7%, referent found rate 28.5%, within-match accuracy 0.849 \(\rightarrow\) When it parses and finds the referent, its classification accuracy is high; the deficit is formatting and under-commitment (average response length 14,429 characters in endless reasoning loops).
Key Findings¶
- Dichotomy of Small-Model Failures: The open-task degradation originates from two distinct pathologies: Binding Failure (exemplified by Gemma-4B and Qwen-27B, where output formats cleanly and extracts entities but defaults to "ambiguous" when gender cues cross clause boundaries) versus Format Failure / Under-commitment (exemplified by Qwen-4B and Ministral, which generate unbounded reasoning without committing to a parseable decision).
- Single-Class Training Trap: Fine-tuning exclusively on authentic ambiguous sentences (FT-GAND) destroys the model's discriminative threshold, resulting in an abstention collapse where almost every instance is predicted as ambiguous. High-quality counterfactual contrastive variants (FT-CONTRA) are strictly mandatory to provide positive commitment boundaries.
- Locality Penalty: Local cues (e.g., GENDER ADJECTIVE \(\ge 0.92\) accuracy across all parseable models) are universally well-resolved. In contrast, cross-clause cues cause dramatic drops, with VOCATIVE ADDRESS exhibiting a 57 pp disparity between Gemma-31B (0.935) and Gemma-4B (0.364).
Highlights & Insights¶
- Symbolic Guardrails for Generative Data: Rather than relying purely on LLM-as-a-judge (which is known to harbor length and position biases), the integration of deterministic post-validators for Levenshtein distance, token parity, and POS stability guarantees strict minimal-pair purity.
- The Closed-to-Open Gap as a Fairness Diagnostic: Demonstrates that reporting closed-task classification accuracy artificially inflates LLM competence; open evaluation uncovers realistic agentic bottlenecks in autonomous entity extraction and cross-clause context linkage.
- Highly Efficient Student Repair: In under five GPU hours of QLoRA training on a single 4-GPU node, a 4B parameter student model is upgraded from an unusable open-task baseline (0.242) to matching a 31B frontier teacher (0.976), delivering an economical solution for production-grade ambiguity detection.
Limitations & Future Work¶
- English-Centric and Genre Constraints: CONTRAGAND is derived from English C4 and OpenSubtitles. In morphologically rich languages (e.g., French, German, Spanish), gender marking is pervasive across articles, adjectives, and participles, precluding direct transfer of the nine-strategy taxonomy.
- Binary Classification Scope: The current taxonomy restricts targets to binary masculine/feminine categories alongside an ambiguous abstention class, omitting non-binary identities and singular "they".
- Downstream Translation Integration: While the benchmark provides a standalone ambiguity detector, verifying its real-time orchestration within end-to-end translation pipelines to suppress downstream stereotyping remains an ongoing investigation.
Related Work & Insights¶
- vs. WinoBias / Winogender / GAP / BUG: Traditional benchmarks pre-define coreferent pairs and evaluate pronoun resolution on hand-crafted templates; CONTRAGAND evaluates natural sentences without overt pronouns and measures open, bottom-up referent discovery.
- vs. Pure-Rule CDA (Lu et al., 2020): Heuristic pronoun-swapping fails on non-pronominal gender devices; CONTRAGAND's neurosymbolic pipeline spans 9 distinct local and cross-clause linguistic mechanisms.
- vs. Self-Instruct / Constitutional AI: Unconstrained self-critique struggles on small models due to verbosity and runaway self-reflection; this work demonstrates that deterministic error signals combined with contrastive supervision provide the necessary constraints for reliable convergence.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first open-task gender ambiguity diagnostic with a rigorous 9-strategy neurosymbolic counterfactual pipeline.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 6 models, complete with funnel decompositions, decoding sweeps, bootstrap CIs, McNemar significance tests, and distillation ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity in diagnosing failure modes, supported by precise terminology and compelling empirical traces.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable, open-source dataset and efficient repair recipe for mitigating gender bias in translation and LLM deployments.