PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.2/
Code: https://github.com/sumin-yu/PopResume
Area: LLM Safety
Keywords: Resume Screening, Causal Fairness, Path-Specific Effects, LLM Auditing, Multimodal Bias
TL;DR¶
Seoul National University researchers introduce PopResume, the first 60.8k population-representative resume dataset grounded in U.S. Census statistics, accompanied by a Path-Specific Effect (PSE) causal auditing framework that decouples legally permissible business necessity from impermissible redlining disparities in LLM and VLM hiring screeners.
Background & Motivation¶
The rapid adoption of Large Language Models (LLMs) and Vision-Language Models (VLMs) in automated resume screening and recruitment decision-making has triggered acute concerns regarding systemic demographic discrimination. Existing auditing benchmarks and corporate compliance evaluations typically focus on aggregate outcome disparitiesโsuch as the EEOC's Four-Fifths ruleโor rely on ad-hoc perturbation protocols, such as manually swapping gender-typical names on a static resume template.
However, this conventional paradigm suffers from two fatal conceptual and legal shortcomings: First, anti-discrimination law (notably Title VII of the U.S. Civil Rights Act of 1964) establishes that an observed demographic disparity in selection rates is not inherently illegal. If a disparity is mediated through legitimate, job-essential qualifications (e.g., years of experience or degree requirements under the "business necessity" doctrine), it is legally defensible. Conversely, if disparity is transmitted through demographic proxy variables (such as applicant names, residential zip codes, or profile photographs under the "redlining" doctrine), it constitutes impermissible discrimination. Aggregate outcome-level metrics are fundamentally blind to these causal pathways. Second, manual attribute injection disrupts the natural joint distributions governing real-world labor markets (such as empirical correlations between age, educational attainment, and years of workforce experience), resulting in unnatural counterfactuals that cannot support valid causal effect estimation.
To bridge this gap, this work reformulates resume auditing through Structural Causal Models (SCMs), grounding applicant distributions in authentic population-level census microdata and decomposing the mediator space into legally distinct channels. Core idea: construct PopResume, a 60,884-resume dataset grounded in U.S. Census Bureau statistics across five occupations, and establish a Path-Specific Effect (PSE) framework that disentangles total hiring disparity into the legally defensible Business-necessity Indirect Effect (BIE) and the legally impermissible Redlining Indirect Effect (RIE), exposing five critical discrimination patterns that escape aggregate metrics.
Method¶
Overall Architecture¶
The PopResume framework consists of two core components: (1) a population-representative generative pipeline grounded in public statistical surveys; and (2) a path-specific causal decomposition protocol evaluating downstream LLM/VLM scoring behavior.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Joint Census Data Harmonization<br/>ACS PUMS + PSID + SSA + Census 2010"] --> B["Empirical Joint Factorization<br/>Protected Attributes X + Qualifications B + Proxies R"]
B --> C["PopResume Multimodal Instantiation<br/>Text-only / Image-only / Image with Headshot"]
C --> D["LLM / VLM Automated Scoring<br/>Job-Description Conditioned Score Y in 0-100"]
D --> E["Path-Specific Effect (PSE) Auditing<br/>Total Effect TE = Direct NDE + BIE + RIE"]
Key Designs¶
1. Factorized Population Distribution Sampling: To reconstruct representative applicant profiles without leaking private individual identities, the authors factorize the joint attribute distribution into four conditional components: $\(P(V \setminus \{Y\} \mid J) = P(X, Z, \text{state}, \text{edu\_level} \mid J) \cdot P(\text{exp\_year} \mid X, Z, \text{state}, \text{edu\_level}, J) \cdot P(\text{first\_name} \mid \text{gender}, \text{age}) \cdot P(\text{surname} \mid \text{race})\)$ where \(X\) represents protected attributes (gender, race), \(Z\) represents exogenous demographic covariates (age), \(B\) represents qualification mediators (education level, years of experience), and \(R\) represents demographic proxies (name, state). Experience years are modeled via an XGBoost regressor trained on density-ratio reweighted Panel Study of Income Dynamics (PSID) data. First names and surnames are conditioned on SSA and 2010 Census distributions. Resumes are instantiated into natural text using strict rule-based deterministic templates, while synthetic profile headshots for VLM evaluation are generated via diffusion models conditioned on demographic parameters.
2. Symmetrical Path-Specific Effect Decomposition (BIE & RIE): Under the Standard Fairness Model (SFM), Total Effect (TE) decomposes into Natural Direct Effect (NDE) and Natural Indirect Effect (NIE). To account for potential mutual dependencies between business qualifications (\(B\)) and proxy variables (\(R\)), the authors define symmetric path-specific indirect effects by averaging over possible intervention orders: - B-specific Indirect Effect (BIE): Isolates the causal effect transmitted exclusively through legitimate job-relevant qualifications (\(B \to Y\)), representing legally defensible variance; - R-specific Indirect Effect (RIE): Isolates the causal effect transmitted through demographic proxy variables (\(R \to Y\)), representing legally impermissible redlining bias.
3. Visual Profile Photo Bias Triggering Protocol: The evaluation tests three distinct input formats: plain structured text, formatted resume images without headshots, and formatted resume images embedding synthetic professional portrait headshots. This isolates whether facial visual cues cause VLMs to bypass textual proxies and directly elevate the Natural Direct Effect (NDE).
Loss & Training¶
The framework operates as an auditing diagnostic, requiring unbiased non-parametric estimation of potential outcomes \(\mathbb{E}[Y_{x, b, r}]\). The authors deploy Double Machine Learning under Unconfoundedness with Cross-fitting (DML-UCA, 5-fold cross-fitting) across all candidate configurations. Standard errors and 95% confidence intervals are derived from 500 bootstrap iterations; an effect is verified as statistically significant if its confidence interval strictly excludes zero.
Key Experimental Results¶
Main Results¶
The authors evaluated 4 text LLMs (Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.2, GPT-4o-mini, Gemini-2.5-Flash-Lite) and 4 VLMs (Qwen2.5-VL-7B-Instruct, InternVL2-8B, GPT-4o, Gemini-2.5-Flash-Lite) across 5 major occupations (Registered Nurses, Construction Laborers, Middle School Teachers, Accountants, etc.), evaluating 60,884 resumes across 120 experimental configurations:
| Model & Evaluation Setup | Target Occupation | Total Effect (TE) | Direct Effect (NDE) | Business Necessity (BIE) | Redlining Proxy (RIE) | Diagnosed Fairness Pattern |
|---|---|---|---|---|---|---|
| Gemini-2.5-Flash-Lite (LLM) | Registered Nurses | -0.12 (โ 0) | -0.85* | +0.42* | -0.31* | Case 2: Masked Cancellation (Apparent parity, deep bias) |
| Mistral-7B-Inst-v0.2 (LLM) | Construction Laborers | +1.85* | +1.22* | +0.35* | +0.28* | Case 1: Direct Discrimination (Explicit labels absent) |
| Llama-3.1-8B-Inst (LLM) | Elementary/Middle Teachers | -0.48* | -0.05 (โ 0) | -0.42* | -0.01 (โ 0) | Case 3: Legitimate Qualification (Legally defensible) |
| Qwen2.5-VL-7B-Inst (VLM) | Registered Nurses | -0.25 (โ 0) | -0.08 (โ 0) | +0.02 (โ 0) | -0.19* | Case 4: Pure Redlining Proxy (Unlawful proxy bias) |
| Qwen2.5-VL-7B-Inst (VLM) | Accountants & Auditors | +0.05 (โ 0) | +0.02 (โ 0) | +0.62* | -0.59* | Case 5: Opposing Mixed Paths (BIE & RIE cancel) |
| GPT-4o (VLM) | Accountants & Auditors | +1.45* | +0.10 (โ 0) | +0.85* | +0.50* | Case 5: Amplified Mixed Paths (BIE & RIE align) |
(Note: Asterisks (*) indicate statistical significance at 95% confidence intervals. Positive values denote scoring favoring male candidates; negative values favor female candidates.)
Impact of Headshot Photographs on Multimodal VLM Auditing¶
Evaluating paired resume configurations (with versus without profile photos) reveals striking direct bias escalation:
| Evaluated Model & Setting | Image Input Format | Total Effect (TE) | Direct Effect (NDE) | Proxy Effect (RIE) | Mechanistic Causal Insight |
|---|---|---|---|---|---|
| Gemini-2.5-Flash-Lite (Accountant) | Resume without Photo | +0.15 (โ 0) | +0.10 (โ 0) | +0.05 (โ 0) | Near-zero effects across all pathways |
| Gemini-2.5-Flash-Lite (Accountant) | Resume with Photo | +1.20* | +1.45* | -0.25* | Direct Effect (NDE) surges, activating explicit bias |
| GPT-4o (Registered Nurse) | Resume without Photo | -0.20 (โ 0) | -0.15 (โ 0) | -0.05 (โ 0) | Disparities remain negligible |
| GPT-4o (Registered Nurse) | Resume with Photo | -0.95* | -0.80* | -0.15* | Facial morphology strongly amplifies gender preference |
Key Findings¶
- Uncovering Five Causal Discrimination Archetypes:
- Case 1 (Direct Discrimination without Explicit Labels): In 49 out of 80 text configurations devoid of explicit demographic markers, models exhibited significant non-zero NDE, proving that LLMs infer protected traits from latent contextual correlations.
- Case 2 (Discrimination Masked by Cancellation): In 6 configurations, total disparity TE was statistically indistinguishable from zero, yet underlying direct (NDE) and indirect (NIE) pathways were substantial and counteracted each otherโmisleading outcome-based audits into declaring the screener "fair."
- Case 3 (Disparities Driven by Legitimate Qualifications): Disparities were entirely explained by BIE without proxy redlining (RIE โ 0), establishing legal compliance under business necessity.
- Case 4 (Discrimination via Demographic Proxies): Disparity propagated purely through RIE (names and locations), constituting textbook unlawful redlining.
- Case 5 (Mixed Mediation Pathways): Occurred in 44% of cases (53/120), where BIE and RIE either cancelled or compounded each other, demonstrating the absolute necessity of pathway decomposition.
- Profile Photos Escalate Direct Visual Discrimination: In 8 out of 20 paired VLM comparisons, incorporating headshot photographs significantly inflated the magnitude of NDE, confirming that visual features provide direct channels for algorithmic bias.
Highlights & Insights¶
- Bridging Legal Jurisprudence and Formal Causality: This work provides the first rigorous mathematical translation of employment anti-discrimination doctrine ("business necessity" vs. "redlining") into quantifiable path-specific causal estimands (BIE and RIE).
- Statistical Fidelity over Toy Perturbations: Rather than substituting isolated tokens on synthetic templates, PopResume grounds multi-attribute relationships in U.S. Census microdata, enabling credible counterfactual estimation without breaking empirical covariate structures.
Limitations & Future Work¶
- Geographic and Societal Specificity: The dataset is grounded in U.S. census and occupational surveys, which reflect American demographic distributions and legal standards. Extending the framework to international contexts (e.g., EU-LFS) represents a vital next step.
- Controlled Textual Synthesis versus Open-Ended Nuance: While rule-based instantiation ensures pristine variable isolation, it does not fully encompass unstructured conversational nuances or non-standard career trajectories observed in human-written resumes.