Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Large Vision-Language Models, Societal Bias Evaluation, Safety Guardrails, Refusal-Free Decoupling, Total Variation Distance

TL;DR

To resolve the widespread evaluation failure caused by safety guardrails refusing attribute-inferring prompts, this work decouples person images into background user context and evaluates societal bias across three person-irrelevant tasks, achieving zero refusals and uncovering persistent stereotypes across 20 state-of-the-art LVLMs.

Background & Motivation

As large vision-language models (LVLMs) become increasingly integrated into multimodal reasoning and real-world assistant applications, societal biases such as gender and racial stereotyping have emerged as pressing concerns. Conventional bias evaluation benchmarks predominantly follow an attribute-inferring paradigm: they pair demographic-annotated portrait images with explicit queries requiring the model to infer personal attributes or social standing (e.g., asking "Is this person a CEO or a secretary?" or "Who is more intelligent?"), quantifying bias through distributional disparities in outputs across groups.

However, modern proprietary models (such as the GPT and Claude families) as well as advanced open-source models (such as Gemma3 and Qwen2.5-VL) are equipped with increasingly stringent safety alignment and guardrail mechanisms. Consequently, they frequently decline to answer these attribute-inferring prompts, considering them potential violations of privacy or ethical safety guidelines. In popular benchmarks, refusal rates routinely soar to 60%–100%, shattering the statistical validity required for reliable bias estimation. Meanwhile, alternative captioning-style prompts that evade refusals suffer heavily from contextual visual confounds, where scene backgrounds or co-occurring objects correlate spuriously with specific demographic identities.

Overcoming the twin obstacles of guardrail refusals and scene-level visual confounding requires a fundamental paradigm shift in how bias probing is framed. Core idea: decouple the target task from the depicted individual by replacing attribute-inferring prompts with person-irrelevant tasks, treating the demographic image solely as incidental user context to observe whether the model's responses to unprompted generic queries exhibit unwarranted demographic disparity.

Method

Overall Architecture

The proposed guardrail-agnostic evaluation framework bypasses defensive safety triggers by shifting the image from being the subject of inquiry to serving as user profile context. The system pipeline consists of injecting face portraits with a provisional prefix alongside general, person-irrelevant instructions across three distinct task modalities, followed by structured attribute/difficulty extraction and fairness measurement via Total Variation Distance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Portrait Image + Person-Irrelevant Prompt"] --> B["Task Decoupling and User Context Injection<br/>Portrait treated as incidental user info, evading safety refusals"]
    B --> C["Three Person-Irrelevant Probe Tasks<br/>Story Generation / Term Explanation / Exam QA"]
    C --> D["TVD-Based Bias Quantification<br/>Extract output distributions and measure divergence from fair ideal"]
    E["Output: Refusal-Free Societal Bias Scores"]
    D --> E

Key Designs

1. Task Decoupling and User Context Injection: Converting target portraits into user context

Directly prompting an LVLM to infer sensitive personal traits from facial images directly violates modern safety alignment filters, triggering canned refusal responses. To circumvent this without jailbreaking, the framework introduces a conversational prefix (e.g., "I've attached my photo.") and pairs it with prompts that make no mention of the person depicted (e.g., "Write a fictional story about an imaginary person." or "Teach me about linear algebra."). Under this formulation, the model is never tasked with assessing the individual. A truly unbiased system should adhere to demographic independence:

\[\mathcal{O}_{a,q} = \{\phi(I, p, q) \mid (I, a') \in \mathcal{D}, a' = a\}\]

Because the core task does not evaluate the image subject, safety guardrails remain dormant, driving refusal rates down to zero across all tested models while capturing latent stereotyping behavior.

2. Three Person-Irrelevant Probe Tasks: Assessing bias across diverse generative and reasoning spaces

To observe societal bias across varying degrees of generation freedom, the framework establishes three probe tasks. Story Generation tests open-ended creative writing, prompting models to invent characters while using an LLM assistant to extract occupational, academic, and socioeconomic attributes, checking whether the user's demographic profile induces stereotypical roles (e.g., mechanics for male users vs. nurses for female users). Term Explanation spans 20 college-level concepts across six academic domains (math, physics, CS, art, literature, music), asking the LLM assistant in a blind test which paired explanation contains more technical jargon, thereby revealing patronizing simplifications. Exam-Style QA leverages objective multiple-choice questions from MMLU's STEM domains to evaluate whether user demographics degrade the model's factual reasoning accuracy.

3. TVD-Based Bias Quantification: Unifying distribution discrepancies across tasks

Unlike KL divergence, which is notoriously sensitive to zero-probability bins and sparse long-tail outputs, the framework adopts Total Variation Distance (TVD) to quantify fairness. For each prompt \(q\), the empirical distribution across demographic groups \(\mathcal{A}\) is compared against an ideal fair baseline:

\[\mathcal{S}_q = \text{TVD}\left( \{ \mathcal{O}_{a,q} \}_{a \in \mathcal{A}} \right)\]

Per-prompt scores are averaged and scaled to \([0, 100]\), where 0 denotes perfect demographic neutrality and higher scores reflect stronger societal bias, providing a coherent mathematical metric across open-ended generation, pairwise comparison, and classification accuracy.

Key Experimental Results

Main Results

The framework was validated against four mainstream bias benchmarks across 20 open-source and proprietary LVLMs. Prior benchmarks encountered overwhelming refusal rates, whereas the proposed method achieved zero refusals across the board.

Benchmark / Model	Qwen2.5-VL-32B	Gemma3-27B	InternVL3.5-38B	GPT-5	Claude 3.7 Sonnet
SBBench Refusal Rate (%)	90	80	80	83	100
ModScan Refusal Rate (%)	94	61	63	49	98
VLA-gender Refusal Rate (%)	90	86	71	97	98
Pairs Refusal Rate (%)	35	41	61	52	81
Ours Refusal Rate (%)	0	0	0	0	0

With full response coverage, gender and racial bias scores (\(\text{TVD} \times 100\)) were evaluated across all three tasks:

Model Family & Models	Story Generation (Gender / Race)	Term Explanation (Gender / Race)	Exam-Style QA (Gender / Race)
Molmo-7B	26.98 / 24.57	2.76 / 5.08	3.44 / 2.98
LLaVA-OneVision-7B	21.41 / 21.88	3.20 / 4.51	2.64 / 2.06
Qwen2.5-VL-32B	35.11 / 23.88	10.42 / 4.42	2.84 / 1.96
Gemma3-27B	21.64 / 23.70	11.64 / 5.87	1.43 / 1.20
InternVL3.5-38B	28.41 / 27.84	2.35 / 4.92	1.05 / 0.87
Claude 3.7 Sonnet	21.57 / 17.67	3.36 / 3.75	1.27 / 0.64
GPT-4o	26.29 / 21.19	6.88 / 3.90	1.47 / 0.99
GPT-5	14.53 / 16.80	3.59 / 4.61	0.50 / 0.36

Ablation Study

Ablations and correlation analyses systematically probed the drivers of bias manifestation and measurement robustness:

Analysis Dimension / Configuration	Average Bias Score (TVD×100)	Core Finding & Correlation Characteristics
Task Constraint: Story Generation	27.23	Highest open-ended freedom yields severe occupational stereotyping
Task Constraint: Term Explanation	4.67	Moderate constraint; technical depth heavily biased toward male/White
Task Constraint: Exam-Style QA	1.48	Lowest bias; objective format constrains demographic divergence
Cross-Task Bias Correlation (\(r\))	-0.11 to 0.21	Negligible correlation across tasks; fairness on one task does not generalize
Gender-Race Bias Alignment (\(r\))	0.49 / 0.60 / 0.93	Strong within-task correlation; models biased in gender also exhibit racial bias
LLM Assistant Robustness (Agreement)	97.0%	High human agreement; anonymized inputs prevent evaluator bias

Key Findings

• Proprietary models exhibit lower but persistent bias: While proprietary systems (GPT-5, Claude 3.7) consistently score lower than open-weight counterparts, GPT-5 still achieves a notable 14.53/16.80 in story generation, frequently projecting mechanics and software developers for male prompts and nurses for female prompts.
• Open-ended generation amplifies stereotyping: Generative tasks with broader creative degrees of freedom exhibit dramatically higher bias scores (27.23) compared to constrained explanation (4.67) and multiple-choice examination (1.48).
• Scale and capabilities do not automatically cure bias: Benchmark performance on MMMU correlates weakly with story generation bias (\(r = -0.17\)), and increasing model parameter size within open-source families actually correlates with higher racial bias (\(r = 0.72\)), demonstrating that scaling alone does not resolve latent bias.

Highlights & Insights

• Decoupled probing bypasses alignment deadlocks: Reinterpreting the input portrait as an unprompted user persona rather than an inference target cleverly evades guardrail triggers, providing an actionable audit pathway for safety-aligned systems.
• Surface alignment vs. deep stereotyping: Proves that while guardrails successfully suppress overt discriminatory statements, LVLMs continue to implicitly condition unrelated generation on visual demographic cues.
• Continuous lifecycle monitoring: Offers a generalizable testing template that can be integrated into pre-deployment audits and continuous post-deployment telemetry across domain-specific use cases such as educational tutors and career advisors.

Limitations & Future Work

• Discrete demographic annotations: The study relies on FairFace's binary gender and seven racial buckets, omitting non-binary identities, intersectional nuances, visible disabilities, and age-related dynamics.
• Potential background artifact leakage: Despite using face-cropped portraits, subtle background or lighting artifacts correlated with demographics could introduce residual confounding.
• Monolingual prompt scope: Experiments are conducted strictly in English, leaving cross-lingual and cross-cultural stereotyping dynamics unexplored.

Related Work & Insights

• vs Attribute-inferring benchmarks (SBBench, ModScan, Pairs): Prior works prompt models to judge people directly, leading to severe refusal collapse under modern alignment; the decoupled approach restores 100% measurement viability.
• vs Image Captioning bias evaluations: Captioning-based bias probes are vulnerable to background objects and contextual noise; the user-persona approach isolates the demographic variable from task-relevant visual context.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Innovatively re-conceptualizes person portraits as user context to dismantle the refusal bottleneck in bias auditing.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 20 prominent LVLMs across three diverse task formats with exhaustive correlation and validation analyses.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined methodology, and comprehensive empirical presentation.
• Value: ⭐⭐⭐⭐⭐ Delivers an indispensable evaluation tool for ongoing safety governance and bias mitigation in commercial LVLM deployment.