Same Person, Different Depiction: Counterfactual Evaluation of Vision-Language Models on Individuals with Limb Deficiencies

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Vision-Language Models, Counterfactual Evaluation, Limb Deficiencies, Model Reliability, Social Bias

TL;DR

This paper introduces InclusiveCFImageBias, a matched-pair counterfactual benchmark (462 pairs, 924 images, 22k+ queries) that holds identity, scene, and activity fixed while toggling visible limb-deficiency cues under ability-neutral prompts, demonstrating that frontier VLMs systematically shift toward higher ratings, more affirmative decisions, and less neutral wording.

Background & Motivation

Vision-language models (VLMs) are increasingly integrated into assistive interfaces, visual chat systems, and automated decision-making pipelines. Users frequently prompt these models to assess candidates, evaluate professionalism, or provide recommendations from images, expecting consistent and objective criteria grounded solely in task-relevant visual evidence. However, individuals with disabilities—who represent roughly 16% of the global population according to WHO estimates—have been almost entirely absent from controlled visual evaluations of multimodal models. While social bias benchmarks have extensively probed demographic dimensions such as race and gender, limb deficiencies remain an unisolated visual axis.

Isolating visual disability bias in the wild is notoriously challenging. Comparing disparate real-world photos introduces confounding factors like clothing, lighting, facial appearance, background scenes, and postures, making it impossible to attribute output differences directly to disability cues. Furthermore, common intuition often presumes that disability signals necessarily trigger negative prejudice or scoring penalties. Yet, when an image query is entirely unrelated to physical mobility or manual dexterity, an arbitrary boost in ratings or an outpouring of patronizing praise still signals significant criteria drift. Such well-meaning over-accommodation or heroization undermines system reliability and distorts representational fairness.

To address this blind spot, this work frames the evaluation of disability representation as a rigorous paired counterfactual stability test. If a query does not ask about physical ability, should a model shift its judgments simply because a residual limb or prosthesis is visible? Core idea: construct InclusiveCFImageBias, a matched-pair benchmark keeping identity, background, and activity invariant via localized inpainting while toggling limb-deficiency cues under ability-neutral prompts, directly quantifying decision drift, score inflation, and framing shifts across proprietary and open-weight VLMs.

Method

Overall Architecture

InclusiveCFImageBias establishes an experimental pipeline designed to attribute model output shifts strictly to visible limb-deficiency evidence. The pipeline consists of sourcing licensed real-world photographs, precise segmentation and cue annotation, diffusion-based localized inpainting to generate counterfactual depictions, multi-stage human verification and blind spot checks, followed by an isolated single-image paired prompting protocol across structured and open-ended queries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real-world Licensed Photos<br/>LDPose CC BY-NC-SA 4.0"] --> B["Cue Segmentation & Tagging<br/>Prosthesis / residual-limb masks"]
    B --> C["Localized Counterfactual Inpainting<br/>Qwen-Image-Edit & original pixel preservation"]
    C --> D["Human Filtering & Blinding Verification<br/>Reject artifacts / face drift; 462 pairs kept"]
    D --> E["Ability-Neutral Multi-Paradigm Prompt Bank<br/>Decision / 1-5 rating / open-ended / control"]
    E --> F["Paired Single-Image Query Protocol<br/>Measure TVD, rating shifts, and regard drift"]

Key Designs

1. Localized Inpainting and Quality Verification: Confounder-Free Counterfactual Pair Generation

To ensure that behavioral drift is strictly attributable to the target visual attribute, candidate images are curated from the CC BY-NC-SA 4.0 licensed LDPose dataset, filtering out extreme occlusions or clinical settings. Annotators draw fine-grained segmentation masks around visible prostheses or residual limbs and document metadata tags. Counterfactual depictions are synthesized using Qwen-Image-Edit, restricted solely to the masked bounding region while keeping all outside pixels identical to the original image. The pipeline generated 2,183 candidate pairs, from which multi-stage manual inspection discarded any samples exhibiting facial deformation, identity drift, or skin tone shifts, leaving 462 verified pairs (924 images). A blinded "spot-the-edit" experiment verified that independent annotators located the edited limb with less than 10% accuracy, verifying the visual naturalness of the counterfactual images.

2. Physical-Ability-Neutral Prompt Bank: Decoupling Social Judgments from Physical Capability

Evaluating bias without confounding requires prompts that genuinely avoid testing physical performance. The authors curated 48 prompts spanning six critical social judgment domains: competence/performance, leadership, teamwork/social coordination, professionalism, communication, and hiring/recommendation. Prompts were vetted to exclude references to physical exertion, running, or manual handling, with 200 instantiated queries independently double-checked to confirm the absence of ability connotations. The prompt bank includes three complementary formats: control prompts (~29% of units, querying unchanged background elements to establish baseline noise), structured decision and rating prompts (~33%, querying Yes/No/Uncertain or 1–5 ordinal scales), and open-ended generation prompts (~38%, probing conversational framing, tone, and implicit social assumptions).

3. Paired Differential Evaluation Protocol: Quantifying Decision Certainty and Regard Drift

During testing, models never receive paired images concurrently; each image (original vs. counterfactual) is queried independently in a separate single-image session under identical prompt phrasing. Paired shifts are defined as \(\Delta = \text{Counterfactual} - \text{Original}\). On structured tasks, stability is measured by mean rating difference \(\Delta\text{Rating}\) with 95% confidence intervals, two-sided sign test significance, and Total Variation Distance (TVD) across categorical decision distributions: $\(\text{TVD} = \frac{1}{2} \sum_{y \in \{\text{Yes}, \text{No}, \text{Uncertain}\}} |P_c(y) - P_o(y)|\)$ For open-ended outputs, the framework computes surface-level lexical similarities (Jaccard and BERTScore F1), directional semantic entailment via Natural Language Inference (NLI), and social sentiment shifts (Regard: positive, neutral, negative) to detect unsolicited praise, pity, or non-neutral evaluative stance shifts.

Key Experimental Results

Main Results

The benchmark evaluated six prominent VLM families spanning open-weight architectures (Qwen3-VL, DeepSeek-VL2, Gemma-3, Ministral-3) and proprietary frontier models (GPT-5, Gemini-2.5) without decoding-parameter tuning.

Table 1: Paired shifts in structured decisions and 1–5 ratings (\(\Delta = \text{Counterfactual} - \text{Original}\))

Model Family & Variant	Control Consistency	\(\Delta\text{Rating}\) (Mean)	95% CI	Sign Test \(p\)-value	\(\Delta\text{Yes}\) (pp)	\(\Delta\text{Unc.}\) (pp)	Decision Drift TVD (pp)
GPT-5	0.822	-0.089	[-0.146, -0.033]	\(3.52 \times 10^{-3}\)	-1.84	+0.76	1.84
Gemini-2.5	0.736	-0.404	[-0.511, -0.298]	\(1.64 \times 10^{-9}\)	-3.86	+3.19	3.86
Qwen3-VL 30B-A3B-Instruct	0.759	-0.205	[-0.252, -0.156]	\(4.81 \times 10^{-18}\)	-6.82	+3.14	6.82
Qwen3-VL 30B-A3B-Thinking	0.738	-0.114	[-0.157, -0.075]	\(6.35 \times 10^{-7}\)	-5.35	+3.11	5.35
Qwen3-VL 8B-Instruct	0.813	-0.375	[-0.436, -0.314]	\(1.90 \times 10^{-38}\)	-2.71	+1.73	2.71
Qwen3-VL 8B-Thinking	0.720	-0.145	[-0.195, -0.094]	\(1.21 \times 10^{-7}\)	-6.93	+8.33	8.33
Gemma-3 12B-it	0.751	-0.180	[-0.221, -0.140]	\(3.48 \times 10^{-22}\)	-3.68	-0.87	4.55
DeepSeek-VL2	0.759	-0.214	[-0.272, -0.153]	\(2.61 \times 10^{-12}\)	-11.90	+13.53	13.53
DeepSeek-VL2-Small	0.754	-0.088	[-0.114, -0.063]	\(3.48 \times 10^{-11}\)	-3.46	+3.46	3.46
Ministral-3 8B-Instruct	0.699	-0.207	[-0.242, -0.169]	\(2.63 \times 10^{-28}\)	-10.20	+8.75	10.20
Ministral-3 8B-Reasoning	0.812	-0.143	[-0.172, -0.115]	\(2.65 \times 10^{-22}\)	-4.98	+4.11	4.98

Note: Negative \(\Delta\text{Rating}\) and negative \(\Delta\text{Yes}\) indicate that the original depiction (visible limb deficiency) receives higher ratings and more affirmative decisions than the intact counterfactual.

Ablation Study

The authors conducted comprehensive validation experiments to rule out confounding variables such as inpainting artifacts, multi-person attention splits, inherent sampling noise, and topic-specific artifacts.

Table 2: Control and ablation configurations against alternative explanations

Experimental Split / Control	Sample Size / Note	\(\Delta\text{Rating}\)	Decision TVD (pp)	Text Dissimilarity / Jaccard	Finding & Attribution Conclusion
Main Benchmark (All Pairs)	462 full matched pairs	-0.151	7.11	0.562 (dissimilarity)	Primary observed depiction shift
Same-Cue Edit Control	Re-inpainted with cue kept visible	-0.152	6.36	—	Shift persists identically; rules out inpainting residue
Single-Person Split	392 single-subject pairs	-0.169	7.73	—	Effect strengthens; rules out multi-person attention ambiguity
Multi-Person Split	70 multi-subject pairs	-0.042	lower	—	Milder effect; confirms single-person scene drives the trend
Repeated Generation Noise	Identical image/prompt repeated	0.122 (variance)	1.81	0.167 (dissimilarity)	Intrinsic sampling noise is far smaller than paired drift (1.81 vs 8.89)

In prompt-topic analyses, all six domains exhibited consistent negative \(\Delta\text{Yes}\) and non-zero TVD (ranging from 6.14 to 13.66 pp), with Professionalism and Competence displaying the strongest shifts, demonstrating that criteria drift is systemic rather than an artifact of isolated prompt wording.

Key Findings

• Pervasive Positive Score Uplift: Rather than penalizing disability, almost all models systematically assigned higher 1–5 ratings when limb-deficiency cues were visible under ability-neutral prompts (e.g., Gemini-2.5 shifted by -0.404, Qwen3-VL 8B-Instruct by -0.375). This exposes a widespread tendency toward over-accommodation or socially desirable positivity.
• Decision Drift from Yes to Uncertain: Counterfactual cue removal predominantly shifted affirmative decisions (Yes) toward hesitation (Uncertain) rather than outright rejection (No). For instance, DeepSeek-VL2 saw an 11.90 pp drop in Yes and a 13.53 pp rise in Uncertain, indicating that disability cues prompt premature decisiveness.
• Reasoning Architecture Dampens Sensitivity: Across model families, instruction-tuned checkpoints exhibited significantly larger rating and regard shifts than their reasoning-oriented or thinking counterparts (e.g., Qwen3-VL 30B-Instruct at -0.205 vs. Thinking at -0.114; Ministral-3 8B-Instruct at -0.207 vs. Reasoning at -0.143), suggesting that structured reasoning steps help suppress superficial cue-driven heuristics.

Highlights & Insights

• First Vision-Side Counterfactual Disability Benchmark: Bridges a major gap in multimodal fairness research by moving beyond race and gender to establish the first controlled vision-grounded benchmark for limb differences.
• Reframing Bias Beyond Penalties: Demonstrates that well-intentioned positive shifts (inflated ratings, unsolicited praise, or patronizing sympathy) represent a severe compromise of criteria stability and model reliability in high-stakes visual evaluations.
• High-Fidelity Inpainting Protocol: Combines localized diffusion-based editing with strict identity preservation and blinded spot-checking, providing an extensible blueprint for auditing other sensitive visual attributes.

Limitations & Future Work

• Admitted Limitations: Due to licensed photo availability, the dataset displays an over-representation of male subjects (66.4%) and White individuals (62.8%), predominantly featuring lower-limb amputations or prostheses, with limited coverage of upper-limb or multi-limb differences.
• Spotted Limitations: The prompts are currently limited to English, leaving open questions about how multilingual and cross-cultural alignments modulate disability representation and over-accommodation; subtle inpainting frequency artifacts could theoretically interact with vision backbones.
• Future Directions: Exploring paired-consistency regularization during post-training, introducing guardrails against unsolicited evaluative framing, and extending the benchmark to multimodal interactions in video and robotic assistance.

Related Work & Insights

• vs PAIRS / SocialCounterfactuals: While earlier counterfactual studies examined demographic biases (gender, race) and focused on toxicity or negative stereotyping, this work is the first to isolate physical limb deficiencies, discovering an unexpected upward scoring bias driven by alignment over-accommodation.
• vs AccessEval / VizWiz: VizWiz benchmarks task success for visually impaired users, and AccessEval examines disability framing in text-only language models. InclusiveCFImageBias provides the missing visual-side testbed evaluating whether models alter non-physical judgments solely due to visible disability cues.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes the first vision-language counterfactual audit for limb differences, uncovering an unexpected positive-shift bias.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across 6 model families with extensive noise controls, NLI semantic evaluation, and regard shifts.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative, precise conceptual taxonomy, and strong empirical grounding.
• Value: ⭐⭐⭐⭐⭐ Offers an essential regression testing tool for building reliable, unbiased, and inclusive multimodal systems.