Skip to content

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ChartBiasBench/ChartBias
Area: Multimodal VLM
Keywords: chart question answering and reasoning, vision-language models, counterfactual bias, algorithmic fairness, multi-agent collaboration

TL;DR

This paper introduces ChartBias, the first benchmark auditing counterfactual bias in vision-language model (VLM) chart interpretation across 820 real-world charts and six sensitive attributes, uncovering pervasive narrative shift, group hallucination, and preference polarity, while proposing a multi-agent decoupling framework that mitigates divergence.

Background & Motivation

Vision-language models (VLMs) are increasingly integrated into real-world applications including automated chart summarization, digital accessibility aids, and executive analytics assistants. In these settings, charts serve as the principal vehicle for communicating high-stakes quantitative information on socio-economic topics such as employment, healthcare equity, migration patterns, and justice. Because model-generated interpretations are fluent, persuasive, and superficially anchored to visual figures, readers routinely perceive them as authoritative, objective truth. However, when a model produces radically divergent interpretations of an identical quantitative chart merely because the referenced demographic group changes, it weaponizes quantitative reasoning to legitimize harmful stereotypes and skew decision-making.

Despite the proliferation of bias studies in multimodal learning, existing benchmarks primarily focus on generic image-text matching or narrow geographic-economic disparities in chart-to-text generation. The chart understanding literature has almost exclusively concentrated on mathematical factual accuracy and visual element parsing, operating under the implicit assumption that factual grounding guarantees fairness. Consequently, there has been no benchmark systematically assessing whether VLMs maintain counterfactual narrative consistency across diverse social groups, nor any audit measuring whether models abstain appropriately when presented with charts lacking group evidence, or whether favorable versus unfavorable trends are systematically mapped onto privileged versus marginalized populations.

To bridge this critical gap, a rigorous counterfactual auditing protocol is required: visual evidence must be strictly held invariant, completely free from visual attribute leakage, such that any shift in narrative tone, emphasis, or framing is directly attributable to the demographic token substitution. Core idea: establish ChartBias, the first benchmark spanning six sensitive demographic attributes to audit VLM counterfactual bias across narrative shift, group hallucination, and preference polarity, while presenting a multi-agent framework that decouples group-neutral chart evidence extraction from group-conditioned narrative generation.

Method

Overall Architecture

The methodological framework comprises two integrated components: the strictly controlled ChartBias counterfactual evaluation pipeline and a four-stage Multi-Agent Counterfactual Mitigation Framework. During evaluation, 820 curated, attribute-neutral charts undergo paired counterfactual prompt substitutions across six sensitive attributes, measuring narrative divergence, unsupported group attribution, and polarity-linked preference across 12 VLMs. During mitigation, the framework isolates visual fact perception from social identity tokens via deterministic masking, extracts shared structured JSON evidence, synthesizes paired group-conditioned narratives, and applies a consistency judge to detect and rewrite unjustified discrepancies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Chart and Group Input<br/>Raw chart image + sensitive prompt"] --> B["Sensitive Attribute Masking<br/>Deterministic neutral substitution"]
    B --> C["Chart Evidence Extraction<br/>Structured factual JSON extraction"]
    C --> D["Group-Conditioned Paired Generation<br/>Symmetric narrative synthesis"]
    D --> E["Counterfactual Consistency Judge & Rewrite<br/>Stance/semantic/sentiment checks"]
    E --> F["Unbiased Chart Interpretation<br/>Preserved factual grounding"]

Key Designs

1. Sensitive Attribute Masking: Severing demographic semantic contamination in visual perception

Standard VLMs process prompts containing demographic terms (e.g., race, socioeconomic status) end-to-end, allowing social priors encoded in pre-training corpora to bias visual feature encoding and initial pattern interpretation before visual reasoning even begins. To eliminate this upstream semantic contamination, the Masking Agent intercepts the user instruction prior to visual reasoning. It deterministically replaces the primary sensitive group token with a neutral placeholder [GROUP] and the comparison counterfactual token with [OTHER_GROUP]. By scrubbing explicit demographic signals, the visual backbone is constrained to process quantitative graphical visual features in a strictly neutral environment.

2. Chart Evidence Extraction: Establishing an objective group-invariant ground-truth anchor

When models generate chart descriptions conditioned directly on demographic identifiers, they frequently hallucinate group-specific trends from generic trendlines. To guarantee that counterfactual interpretations rest upon identical empirical foundations, the Evidence Agent receives the masked prompt along with the visual chart image. It is explicitly instructed to abstain from guessing social groups or offering subjective evaluations, outputting instead a compact, strictly typed JSON object detailing chart type, coordinate axes, major data points, extreme values, inflections, and overall directional patterns. This structured evidence serves as an immutable, factual substrate for all subsequent text generation.

3. Group-Conditioned Paired Generation: Symmetric perspective synthesis anchored on shared evidence

Once objective facts are isolated, the challenge shifts to tailoring narratives to target perspectives or audiences without reviving biased reasoning paths. The Generation Agent takes the extracted JSON evidence and synthesizes paired responses for target group \(g_a^{(1)}\) and counterfactual group \(g_a^{(2)}\) in parallel. The model is constrained to use only the validated chart facts, treating the injected demographic label strictly as a framing reference or perspective anchor rather than as an empirical measurement entity. By enforcing identical evidence ordering and structural lengths across the pair, the generation ensures structural and factual symmetry.

4. Counterfactual Consistency Judge & Rewrite: Multi-dimensional alignment auditing and closed-loop correction

Even when anchored to shared facts, generative models can introduce subtle divergences through affective tone, evaluative framing (e.g., describing a plateau as "stagnation" for one group versus "stability" for another), or causal conjectures. The Consistency Judge operationalizes a hybrid validation protocol: after re-masking group terms, it evaluates stance agreement, sentence embedding cosine similarity via all-MiniLM-L6-v2, and sentiment score divergence across the paired texts. If any metric breaches a pre-calibrated tolerance threshold (such as semantic dissimilarity exceeding 0.10 or discordant stance labels), a specialized LLM rewrite agent is invoked to align the paired drafts into shared evaluative bounds while strictly conserving the underlying quantitative evidence.

Key Experimental Results

Main Results

The authors evaluated 12 frontier closed-source and open-source vision-language models across 155,484 generated responses on the ChartBias benchmark. The primary evaluation encompasses Narrative Shift (measured by mean semantic dissimilarity Diss. and the percentage of response pairs exceeding the 0.10 threshold under counterfactual swaps) alongside Group Hallucination (measured by abstention rate Abs. when queried on attribute-neutral charts).

Model Race Diss. โ†“ Race %>0.1 โ†“ Income Diss. โ†“ Income %>0.1 โ†“ Religion Diss. โ†“ Religion %>0.1 โ†“ Immigration Abs.(%) โ†‘
Closed Frontier Models
GPT-4o 0.15 73.98 0.15 72.90 0.17 85.29 0.00
GPT-5 0.21 89.59 0.20 86.37 0.22 95.17 1.50
Gemini-3-Flash 0.16 84.97 0.17 87.15 0.22 92.96 0.00
Claude-4-Sonnet 0.02 8.77 0.03 9.82 0.25 86.52 0.00
Open-Source Models
Qwen3-VL-2B 0.16 79.60 0.15 71.26 0.18 82.29 0.00
Qwen3-VL-4B 0.14 70.11 0.13 70.00 0.20 82.29 0.00
Qwen3-VL-8B 0.14 69.29 0.13 60.89 0.18 74.25 0.00
Qwen3-VL-32B 0.11 56.92 0.12 61.20 0.14 58.55 0.00
DeepSeek-VL-7B 0.30 89.98 0.29 87.85 0.17 87.55 0.00
DeepSeek-VL-1.3B 0.41 98.94 0.30 91.00 0.21 90.95 0.00
LLaVA-7B 0.22 88.18 0.21 87.56 0.23 89.13 0.00
Llama-3.2-11B-Vision 0.11 54.21 0.12 53.69 0.15 59.93 0.00

Ablation Study

The authors evaluated the multi-agent mitigation baseline and ablated its primary constituents across both GPT-4o and Gemini-3-Flash to assess their impact on reducing counterfactual narrative shifts.

Config Race Diss. โ†“ Race %>0.1 โ†“ Income Diss. โ†“ Income %>0.1 โ†“ Age Diss. โ†“ Age %>0.1 โ†“ Note
GPT-4o Base 0.15 73.98 0.15 72.90 0.14 57.10 Standard direct prompting
GPT-4o + Multi-Agent Mitigation 0.06 16.54 0.05 16.22 0.04 16.67 Full multi-agent pipeline
Gemini-3-Flash Base 0.16 84.97 0.17 87.15 0.08 20.63 Standard direct prompting
Gemini-3-Flash + Multi-Agent Mitigation 0.04 3.26 0.01 0.68 0.04 3.47 Full multi-agent pipeline
w/o Masking (Gemini-3-Flash) 0.08 23.05 0.04 3.93 0.06 26.44 Omits deterministic group term masking
w/o Refinement Judge (Gemini-3-Flash) 0.09 32.53 0.11 41.50 0.09 33.17 Omits consistency checks and rewrite loop

Key Findings

  • Narrative shift is pervasive across model families: Under identical visual evidence, replacing only demographic group tokens drives substantial divergence in explanations; nearly all models fail Two One-Sided Tests (TOST) under the 0.10 practical equivalence bound. Open-source models exhibit pronounced shifts (e.g., DeepSeek-VL-1.3B reaches 0.41 dissimilarity on race with 98.94% of pairs exceeding threshold), but frontier closed models also degrade substantially (GPT-5 scores 0.21 on race and 0.22 on religion).
  • Models almost universally exhibit group hallucination: When presented with charts devoid of demographic identifiers and asked which group is depicted, models should abstain. Instead, virtually all models abstain near 0.00% of the time, defaulting to confident but baseless demographic attributions (e.g., LLaVA-7B attributes 76.03% to male, and GPT-4o attributes 79.39% to citizen).
  • Preference polarity couples trend valence with social hierarchy: Group hallucinations are systematically correlated with visual chart polarity. In positive charts, GPT-4o selects low-income in 58.74% of instances, but surges to 87.01% in negative charts. Conversely, White is selected in 85.00% of positive charts but drops to 50.30% in negative charts, proving models systematically link favorable trends to privileged groups.

Highlights & Insights

  • First comprehensive counterfactual benchmark for chart interpretation: ChartBias establishes an indispensable fairness benchmark spanning six sensitive dimensions, providing the community with 820 certified charts and over 8,638 counterfactual pairs to audit multimodal bias.
  • Exposing the peril of objective data as a veneer for subjective bias: The findings demonstrate that high mathematical accuracy and fluent chart citation do not preclude deep demographic prejudice; models subtly alter affective framing while citing identical data points, producing a deceptive appearance of objectivity.
  • Plug-and-play decoupled multi-agent mitigation: By separating perceptual evidence gathering from group framing and closing the loop with automated consistency rewriting, the multi-agent framework reduces narrative shift by over 70% without requiring costly model retraining.

Limitations & Future Work

  • Binary group simplification: To control combinatorial complexity and enable rigorous pairwise counterfactual contrasts, the study models sensitive attributes via binary pairs (e.g., White vs. Black, High- vs. Low-income), omitting intersectional and continuous demographic spectra.
  • English and Western data focus: The curated charts originate primarily from OECD, Statista, VisText, and Pew Research in English-language settings, leaving cross-lingual and culturally diverse chart conventions unaddressed.
  • Subtle residual affective bias: While the multi-agent framework effectively suppresses gross factual divergences, qualitative inspection reveals subtle residual differences in rhetorical tone, severity emphasis, and evaluative framing.
  • vs VisText (Tang et al., 2023) / Chart-to-Text (Obeid & Hoque, 2020): Existing chart-to-text benchmarks evaluate factual coverage, semantic overlap, and data extraction accuracy, lacking sensitive attribute tags and counterfactual consistency checks; ChartBias introduces structured fairness audits to the domain.
  • vs Mahbub et al. (2025b): While Mahbub et al. investigated geo-economic bias in chart generation, their study was restricted to a single geographic attribute and did not evaluate group hallucination or polarity correlation; ChartBias generalizes counterfactual evaluation across six diverse attributes and three failure modes.

Rating

  • Novelty: โญโญโญโญโญ First benchmark and systematic evaluation protocol for counterfactual bias and group hallucination in chart-to-text generation.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated 12 diverse models spanning open and proprietary ecosystems across 155,000+ responses with rigorous statistical hypothesis testing.
  • Writing Quality: โญโญโญโญโญ Well-structured narrative with crisp problem formulation, insightful qualitative case studies, and transparent ablation reporting.
  • Value: โญโญโญโญโญ Vital asset for developing fair, accountable, and transparent multimodal decision-support and data reporting systems.