VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models¶

Conference: ACL2026
arXiv: 2505.22897
Code: https://github.com/chahatraj/Vignette
Area: multimodal_vlm
Keywords: VLM bias evaluation, social stereotypes, VQA benchmark, synthetic images, multimodal fairness

TL;DR¶

VIGNETTE constructs a VQA bias evaluation benchmark featuring over 30M synthetic paired images. By employing four categories of questions—factuality, perception, stereotyping, and decision-making—it reveals that VLMs associate identity cues with activity contexts and social hierarchies, resulting in fine-grained and occasionally contradictory biases.

Background & Motivation¶

Background: While LLM bias evaluation is relatively mature, VLM bias is more complex. Models must not only process textual identity labels but also infer social meaning from appearance, clothing, activities, scenes, and interpersonal comparisons in images. In real-world applications, VLMs may be used for image filtering, content generation, candidate selection, or decision support, making it critical to understand how visual inputs activate biases.

Limitations of Prior Work: Existing VLM bias benchmarks often concentrate on portraits and gender-occupation associations (e.g., "female nurse, male doctor"). This setup is too narrow, lacks activity context, and fails to test whether models infer latent social attributes like ability, morality, status, or suitability for specific roles from identity cues. Furthermore, many evaluations examine identities in isolation, ignoring how relative comparisons between two identities appearing side-by-side can amplify bias.

Key Challenge: Bias evaluation requires covering a vast combination of identities, activities, and social attributes. However, real-world images struggle to systematically cover these dimensions simultaneously. Relying solely on text or headshots fails to approximate the actual behavior of VLMs in contextualized visual inputs.

Goal: The authors aim to build a large-scale, controllable, and contextualized VQA benchmark covering 8 social identity dimensions, 75 activity categories, and 4 evaluation paradigms. This benchmark addresses whether models make factual errors, show biased capability inferences, activate trait stereotypes, and whether these biases influence decision-making.

Key Insight: The paper introduces the Spontaneous Stereotype Content Model (SSCM) from social psychology. It deconstructs social traits into dimensions such as ability, sociability, morality, agency, politics, and status, which are then mapped to VQA questions and role-selection tasks.

Core Idea: By using controllable synthetic images and paired VQA, the study advances VLM bias evaluation from merely "identifying an identity" to understanding "how models compare, infer, and choose between different identities within a visual context."

Method¶

VIGNETTE's approach is not to train a debiasing model but to design an evaluation environment that systematically exposes bias. It first generates identity-activity images, then horizontally stitches two individuals into a paired scene, and finally poses questions at different levels based on the same visual input. This allows the same set of identities to be examined for factual recognition, capability attribution, trait attribution, and role selection.

Overall Architecture¶

Data construction begins with identities and activities. Identities are aggregated from 93 Stigmas, CrowS-Pairs, StereoSet, and HolisticBias, resulting in 167 identity descriptors after deduplication, covering eight dimensions: ability, age, gender, nationality, physical traits, race/ethnicity/color, religion, and socioeconomic status. Activities are derived from time-use theory (necessary, contracted, committed, and free time), totaling 75 visually representable activities.

Image generation utilizes FLUX. Single-person prompts follow the format "An [identity] engaged in [activity], with their face visible," with explicit male/female versions generated to prevent models from introducing inherent gender imbalances. To ensure quality, the authors first generate single images and then stitch them horizontally with slightly blurred boundaries to create paired images for Identity Contrast, Activity Contrast, and Identity-Activity Contrast.

During evaluation, images are fed into VLMs, with Outlines used to constrain outputs to valid options. Questions are divided into four categories: factuality (checking person and activity recognition); perception (whether the model perceives an identity as more struggling, more skilled, more enjoying, or more hating an activity); stereotyping (using non-activity portraits to examine social trait attribution); and decision making (using role-selection questions to observe if bias impacts downstream choices).

Key Designs¶

Activity-Grounded Identity Image Generation:
- Function: Transitions bias evaluation from static headshots to placing identities within real-world tasks and activity contexts.
- Mechanism: Each visually representable identity is combined with 75 activities to generate images, with separate versions for male and female. Activities span cooking, programming, teaching, gardening, praying, playing chess, playing guitar, etc. Paired images are only combined within the same bias dimension to avoid confounding variables between distinct dimensions like age and religion.
- Design Motivation: Many social stereotypes only manifest when considering "who is doing what." Testing whether a model deems a group more suitable for programming or childcare requires activity-grounded visual inputs.
Four-Paradigm VQA Probes:
- Function: Deconstructs bias into a continuous chain from low-level recognition to high-level decision-making.
- Mechanism: Factuality asks "What is someone doing?" or "Who is doing X?"; Perception asks "Who is struggling more/better at/enjoying/hating?"; Stereotyping uses high/low valence word pairs from SSCM (e.g., honest/dishonest, competent/incompetent); Decision Making asks "Who should be selected for X role?".
- Design Motivation: Accuracy alone cannot distinguish if a model misidentified an image or correctly identified it but made a biased inference. These four paradigms link factual errors, capability assumptions, trait stereotypes, and actual selections.
Relative Comparison and Multi-Metric Bias Measurement:
- Function: Captures how the "presence of others" alongside an identity influences model selection.
- Mechanism: Selection Frequency tracks the proportion of times an identity is chosen; Log-Odds measures the over-selection of an identity in an activity; PairComp compares frequency changes when identity \(i_1\) appears with \(i_2\); Polarity Score measures the preference for positive vs. negative traits by subtracting choice rates.
- Design Motivation: Social bias is often relative. An identity might not be disparaged in isolation but may be systematically judged as less capable or lower status when compared side-by-side with another.

Loss & Training¶

This work does not train new models. Evaluated models include LLaVA-1.6-7B, LLaMA-3.2-11B-Vision-Instruct, and DeepSeek-VL2-4.5B. Outputs are converted to discrete choices via constrained decoding. Quality is ensured by two graduate students manually evaluating 1,200 generated images for identity clarity, activity correctness, and the absence of ambiguous features.

Key Experimental Results¶

Main Results¶

VIGNETTE demonstrates significantly broader coverage than existing VLM bias datasets and analyzes systematic biases across multiple VLMs. A core finding is that despite model differences, they exhibit remarkably stable bias structures in perception and decision-making tasks.

Benchmark	Image Type	Data Scale	Bias Scope	Activity Context	Evaluation Task
Existing synthetic	Single synthetic	48K images	9 types + 2 cross	No	Open/Closed QA
Existing race-gender	Single real	700 curated	race × gender × occ	No	MC, desc, completion
Existing trait/occ	Single real	~10K images	gender × traits/skills	Explicitly filtered out	MC classification
VIGNETTE (Ours)	Paired synthetic	30M+ images	8 dimensions × 6 traits	75 activities	Fact., Perc., Stereo., Decision

Evaluation Dimension	Key Observation	Model Trends	Implication
Factuality	Better recognition for socially dominant identities and high-visibility activities	LLaVA-1.6 strongest; DeepSeek-VL2 weaker in SES and religion grounding	Identification errors are themselves identity-biased
Perception	Disabled, old, Middle Eastern, and Native American identities more often judged as "struggling"	Perception scores for most models fall within the 40%-50% range	VLMs infer ability and preference from identity cues
Stereotyping	Traits like morality, status, and sociability are unevenly associated	LLaMA-3.2 better on age/race but stereotypical elsewhere; LLaVA-1.6 poorer across most	Bias occurs in abstract social traits, not just occupations
Decision Making	Healthy, young, traditionally attractive, and mainstream identities preferred	Similar overarching patterns across models despite identity-level variations	High-level choices inherit and restructure low-level stereotypes

Ablation Study¶

Configuration	Key Metric	Description
Identity Clarity	Identity Depicted: 86.2% agreement, Cohen's kappa 0.48	Identification is generally usable, though some visual representations are limited
Activity Clarity	Activity Depicted: 91.2% agreement, kappa 0.82	High activity generation quality supports activity bias evaluation
Lack of Ambiguity	Ambiguous Features=No: 88.7% agreement, kappa 0.94	Majority of images lack significant confounding factors
Overall Distinguishability	951/1200 pairs distinguishable, 88.23% agreement, kappa 0.81	Overall quality supports large-scale VQA evaluation
Prompt Stability	Match rates: Factuality (63-68%), Perception (59-65%), Stereotyping (70%), Decision (66%)	Variations exist by phrasing, but core trends are not induced by a single prompt
Real-vs-Synthetic (PATA)	Mean signed delta 0.0347 pp, MAE 2.9347 pp, RMSE 9.1973 pp	Trends are generally similar between synthetic and real, though local differences can reach 50 pp

Key Findings¶

Factuality is not neutral: Models are more prone to errors for certain identity-activity combinations, indicating that bias analysis must account for visual grounding failures.
Pairwise framing amplifies differences: The probability of choosing an identity changes when it is paired with different counterparts—a scenario closer to real-world comparison than single-image evaluation.
Biases in perception and decision-making are stable: While models show variance in factuality and stereotypes, they exhibit persistent biases in how they perceive effort/skill and make final selections.
Explanation experiments: Analysis of LLaMA-3.2 in a "chef hiring" scenario showed greater attention to male faces and bodies, suggesting bias stems not only from text decoding but also visual attention allocation.

Highlights & Insights¶

VIGNETTE’s primary value lies in framing bias evaluation as a "social inference chain"—querying factuality, then ability, traits, and finally decisions for the same image to observe how bias propagates.
The pairwise design is highly insightful. Fairness is often not about how a model describes one group in isolation, but who it deems more competent, moral, or suitable when two candidates are presented.
Utilizing SSCM to organize stereotypes is more granular than simple occupational bias. Dimensions like morality, agency, and status expose hidden biases that simple category balancing might miss.
The use of synthetic data is disciplined. By stitching single-person images rather than generating complex multi-subject scenes, the authors minimize failures in multi-subject generation that could confound results.

Limitations & Future Work¶

While controllable, synthetic images do not fully represent real social scenarios, and biases inherent in FLUX may carry over into VIGNETTE.
Inclusion is restricted to visually representable identities, excluding important but "invisible" or sensitive attributes like mental health status, sexual orientation, or nuanced cultural identities.
Horizontal stitching, while good for control, is not a natural photographic scenario; models might exhibit sensitivity to boundaries or left/right positioning.
The reliance on multiple-choice VQA is efficient for statistics but limits open-ended explanation. Bias in real applications may manifest in long-form text or image generation.
Social identities and activities require cross-cultural calibration. Certain traits or roles carry different meanings across cultures; future work should expand to multi-region and multi-lingual annotations.

vs. gender-occupation VLM bias: Traditional evaluations focus on limited roles; VIGNETTE expands to 8 identity dimensions and 75 activities across 6 social traits.
vs. portrait-based benchmarks: Portrait tests show identity-trait links but lack activity context. VIGNETTE tests the interaction of "who" and "doing what."
vs. text-only datasets: Text datasets test linguistic priors but cannot assess how visual cues trigger bias. VIGNETTE finds that visual inputs can either increase or decrease selection rates for certain identities compared to text-only baselines.
Implications: Debiasing VLMs should involve inspecting visual encoders and cross-modal attention, not just filtering final outputs, to understand where identity cues are transformed into social evaluations.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Integrates social psychology, pairwise visual scenes, and four-tier VQA tasks effectively.
Experimental Thoroughness: ⭐⭐⭐⭐ Strong data scale and multidimensional analysis, though specific values are sometimes scattered between figures and the appendix.
Writing Quality: ⭐⭐⭐⭐ Clear motivation and task design, though some sections are dense due to the breadth of identity categories.
Value: ⭐⭐⭐⭐⭐ Highly valuable for VLM fairness evaluation, social inference analysis, and multimodal safety research.