Skip to content

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/compling-wat/Chartographer
Area: Multimodal VLM
Keywords: Counterfactual Evaluation, Vision-Language Models, Chart-to-Code Reverse Engineering, Shortcut Learning Diagnostics, Executable QA Logic

TL;DR

CHARTOGRAPHER reverse engineers chart QA samples into semantic data and executable plotting code, validates reconstruction fidelity through iterative self-improvement and human-in-the-loop audits, and programmatically renders counterfactual variants with executable answer logic, revealing that frontier VLMs frequently fail to generalize once visual evidence changes despite answering the original chart correctly.

Background & Motivation

Charts serve as a dense, structured visual language for communicating quantitative relationships, trends, and comparisons across scientific literature, financial reports, and dashboards. Accurately answering questions grounded in charts requires vision-language models (VLMs) to read fine-grained text labels, trace coordinate systems, bind visual encodings to legends, aggregate multi-series values, and execute arithmetic and logical reasoning over visual evidence. As frontier multimodal models are increasingly deployed to automate data analysis, chart question answering (Chart QA) has become an indispensable benchmark for probing general multimodal reasoning capabilities.

However, existing chart understanding benchmarks predominantly evaluate models on static, fixed chart-question-answer triples. This evaluation paradigm obscures whether a model genuinely perceives and reasons over visual marks or merely exploits shortcutsโ€”such as linguistic regularities in the question prompt, canonical layout heuristics, or parametric knowledge memorized during pre-training from web crawls. In conventional static benchmarks, a VLM can easily produce the correct answer through guessing or memorization without reading the chart. What remains fundamentally untested is whether model predictions remain faithfully grounded when the underlying visual evidence changes while the semantic task is held constant.

The fundamental hurdle in testing counterfactual chart reasoning at scale has been the difficulty of altering chart content without either degrading visual quality or requiring prohibitive human re-annotation. Prior diagnostic attempts have either relied on purely textual hypotheticals that leave the image untouched or applied superficial image perturbations like Gaussian blur and color jittering that do not test semantic evidence updating. Core idea: reverse engineer charts into decoupled semantic data and executable plotting code, generate seed-controlled counterfactual charts that alter answer-relevant data while preserving visual schema, and recompute ground-truth answers via executable QA logic to construct chart-question families that strictly isolate visual reasoning from shortcut learning.

Method

Overall Architecture

CHARTOGRAPHER introduces the chart-question family as its primary evaluation unit, comprising the original chart, a base reconstruction rendered from reverse-engineered code with unaltered data, and ten counterfactual variants generated under controlled data seeds. The end-to-end framework operates across four sequential stages: chart-to-code reverse engineering, iterative self-improvement with human validation, counterfactual variant data generation, and executable QA logic generation. The base reconstruction serves as an experimental control verifying that the task survives re-rendering, while the counterfactual variants provide the causal intervention probing whether model predictions adapt to updated visual evidence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Chart QA Sample<br/>original image + question + gold answer"] --> B["Chart Reverse Engineering and Code Decoupling<br/>VLM extracts semantic data and make_figure function"]
    B --> C["Iterative Self-Improvement and Human Validation<br/>multi-turn visual audit and task preservation check"]
    C --> D["Seed-Controlled Counterfactual Data and Chart Rendering<br/>preserving chart schema to generate 10 data variants"]
    D --> E["Executable Logic Driven Dynamic QA Generation<br/>Python functions compute variant answers and adapt question"]
    E --> F["Chart-Question Family Evaluation Suite<br/>evaluates OA / RA / VA / RVC / CVA and CU / NU / SP"]

Key Designs

1. Chart Reverse Engineering and Code Decoupling: Parameterizing visual pixels into reproducible assets

Directly editing rasterized chart images introduces visual artifacts, edge smearing, and typography mismatches that confound downstream visual evaluation. To enable precise, programmatic control over chart data, CHARTOGRAPHER employs a VLM-driven reverse engineering pipeline that maps the source image into two strictly decoupled components: structured semantic chart data and an executable Python plotting function make_figure(data, savepath=None). The semantic data object records entities, numerical series, axis bounds, categorical labels, groupings, and domain orderings. In parallel, the plotting code encapsulates visual encodings, color palettes, typography, legend placements, and multi-panel layouts. Decoupling numerical content from visual rendering logic ensures that the underlying data can be varied arbitrarily while fully preserving the original chart schema and visual presentation.

2. Iterative Self-Improvement and Human Validation: Guaranteeing task preservation and high visual fidelity

Single-pass chart-to-code synthesis frequently encounters minor discrepancies in tick frequencies, legend placement, or font hierarchy. If a reconstruction inadvertently obscures critical visual evidence, subsequent evaluation becomes ungrounded. To prevent reconstruction artifacts from corrupting the evaluation, the framework implements a closed self-improvement loop: the rendered reconstruction is automatically compared against the source image, and the model diagnoses visual mismatches, formulates a revision plan, and refines the code and data across up to five iterative cycles. Reconstructions passing automated checks undergo expert human-in-the-loop review to verify that all task-relevant graphical marks, axis limits, and legend associations remain strictly preserved and answerable. Only high-fidelity reconstructions that retain original task difficulty are accepted into chart-question families.

3. Seed-Controlled Counterfactual Data and Chart Rendering: Deep semantic interventions on visual evidence

To rigorously test whether models genuinely re-ground their visual reasoning rather than relying on shallow heuristics, counterfactual generation must execute meaningful interventions on answer-relevant data points. For every validated chart, CHARTOGRAPHER generates a deterministic Python generator generate_data(data_template, seed). Driven by seeds 0โ€“9, the generator alters values, trajectory intersections, peak coordinates, and category rankings while strictly enforcing template data schemas, physical ranges, and monotonic domain constraints. Rather than applying superficial random noise, these seed-driven regimes induce structural changes in the visual relationships (e.g., swapping the longest bar or inverting a trendline), producing ten distinct counterfactual chart images that require novel visual reasoning pathways.

4. Executable Logic Driven Dynamic QA Generation: Zero-hallucination automated annotation

Manually re-annotating ground-truth answers for thousands of synthetic counterfactual variants is prohibitively expensive and prone to human calculation error. CHARTOGRAPHER circumvents manual re-annotation and LLM hallucination by constructing deterministic executable QA logic consisting of adapt_question(data) and generate_answer(data). The generate_answer module computes the exact target answer directly from the altered data object using programmatic operations (sorting, filtering, thresholding, or arithmetic aggregation), guaranteeing 100% mathematical accuracy. Meanwhile, adapt_question preserves the source question verbatim, making only minimal grammatical adjustments when modified data values would otherwise render the prompt syntactically inconsistent.

Evaluation Metrics and Behavioral Diagnostics

To disentangle baseline competence, reconstruction sensitivity, and counterfactual robustness, the framework defines five quantitative metrics and three qualitative update outcomes: - Original Accuracy (OA): Accuracy on the original benchmark charts: $\(OAm,d = \frac{1}{|F_d|} \sum_{i \in F_d} \mathbf{1}[\hat{y}^o_{m,i} \equiv y^o_i]\)$ - Reconstruction Accuracy (RA): Accuracy on base reconstructions with unaltered data, serving as a control for code re-rendering fidelity. - Variant Accuracy (VA): Average accuracy across all seed-controlled counterfactual variants: $\(VAm,d = \frac{1}{|F_d|} \sum_{i \in F_d} \frac{1}{|V_i|} \sum_{j \in V_i} \mathbf{1}[\hat{y}^v_{m,ij} \equiv y^v_{ij}]\)$ - Relative Variant Change (RVC): Normalized sensitivity to counterfactual alteration, defined as \(RVC_{m,d} = 100 \times \frac{VA_{m,d} - OA_{m,d}}{OA_{m,d}}\). - Conditional Variant Accuracy (CVA): The primary measure of counterfactual generalizability, evaluating variant accuracy strictly conditioned on families where model \(m\) solved the original chart correctly (\(S_{m,d} = \{i \in F_d : \hat{y}^o_{m,i} \equiv y^o_i\}\)): $\(CVAm,d = \frac{1}{\sum_{i \in S_{m,d}} |V_i|} \sum_{i \in S_{m,d}} \sum_{j \in V_i} \mathbf{1}[\hat{y}^v_{m,ij} \equiv y^v_{ij}]\)$ - Update Outcome Diagnostics: Partitioning variant predictions following original-chart success into three mutually exclusive behaviors: 1. Correct Update (CU): The model correctly adapts to the changed visual evidence and predicts the new ground-truth answer. 2. Stale Prediction (SP): The model fails to re-ground, persisting with its original prediction which is now incorrect on the variantโ€”a primary signature of memorization or rigid shortcut reliance. 3. Noisy Update (NU): The model detects visual change and alters its answer, but produces an incorrect prediction due to faulty visual tracking or numerical miscalculation.

Key Experimental Results

Main Results: Cross-Benchmark Performance, Fidelity, and Variant Sensitivity

The framework was evaluated across 440 chart-question families derived from ChartQA, CharXiv, and ChartMuseum, benchmarking 15 frontier proprietary and open-source instruction-tuned VLMs.

Model Category / Model ChartQA (OA / RA / VA / RVC) CharXiv (OA / RA / VA / RVC) ChartMuseum (OA / RA / VA / RVC)
Proprietary Models
Claude Haiku 4.5 0.907 / 0.933 / 0.878 / -3.2% 0.541 / 0.608 / 0.574 / +6.2% 0.472 / 0.451 / 0.357 / -24.3%**
Claude Sonnet 4.6 0.927 / 0.933 / 0.905 / -2.3% 0.662 / 0.750 / 0.697 / +5.2% 0.690 / 0.704 / 0.541 / -21.6%***
Gemini 2.5 Flash 0.933 / 0.933 / 0.891 / -4.5%** 0.608 / 0.669 / 0.647 / +6.4% 0.641 / 0.606 / 0.493 / -23.1%***
Gemini 2.5 Pro 0.947 / 0.927 / 0.897 / -5.2%** 0.682 / 0.750 / 0.726 / +6.3% 0.732 / 0.704 / 0.529 / -27.8%***
GPT-4.1 0.927 / 0.900 / 0.869 / -6.3%** 0.547 / 0.574 / 0.534 / -2.3% 0.507 / 0.549 / 0.392 / -22.6%**
GPT-4.1 mini 0.880 / 0.900 / 0.861 / -2.1% 0.412 / 0.622 / 0.476 / +15.6% 0.528 / 0.528 / 0.446 / -15.5%*
GPT-4o 0.860 / 0.887 / 0.841 / -2.2% 0.432 / 0.426 / 0.389 / -10.2% 0.345 / 0.366 / 0.287 / -16.7%
GPT-5.4 0.927 / 0.947 / 0.916 / -1.2% 0.770 / 0.757 / 0.803 / +4.3% 0.704 / 0.718 / 0.554 / -21.3%***
GPT-5.4 mini 0.940 / 0.940 / 0.917 / -2.5% 0.676 / 0.696 / 0.696 / +3.0% 0.486 / 0.514 / 0.425 / -12.5%
Proprietary Average 0.916 / 0.922 / 0.886 / -3.3% 0.592 / 0.650 / 0.616 / +3.8% 0.567 / 0.571 / 0.447 / -20.6%
Open-Source Models
Gemma 4 E4B IT 0.860 / 0.853 / 0.839 / -2.5% 0.365 / 0.439 / 0.411 / +12.6% 0.289 / 0.261 / 0.251 / -12.9%
InternVL3-8B 0.847 / 0.853 / 0.816 / -3.6% 0.311 / 0.318 / 0.304 / -2.2% 0.268 / 0.239 / 0.223 / -16.6%
LLaVA-OneVision-Qwen2-7B 0.707 / 0.680 / 0.612 / -13.4%** 0.250 / 0.189 / 0.196 / -21.6% 0.190 / 0.211 / 0.164 / -13.7%
Pixtral-12B 0.827 / 0.787 / 0.767 / -7.3%* 0.372 / 0.365 / 0.317 / -14.7% 0.246 / 0.254 / 0.211 / -14.6%
Qwen2.5-VL-7B-Instruct 0.840 / 0.880 / 0.790 / -6.0% 0.338 / 0.372 / 0.343 / +1.6% 0.275 / 0.211 / 0.223 / -18.7%
Qwen3-VL-8B-Instruct 0.907 / 0.907 / 0.882 / -2.7% 0.480 / 0.568 / 0.506 / +5.5% 0.380 / 0.423 / 0.339 / -10.7%
Open-Source Average 0.831 / 0.827 / 0.784 / -5.9% 0.352 / 0.375 / 0.346 / -3.1% 0.275 / 0.267 / 0.235 / -14.5%

(Note: Significance stars on RVC denote two-sided paired sign-flip permutation tests over chart-question families: *p < .05, **p < .01, ***p < .001)

Diagnostic Analysis: Update Behaviors and Reasoning Demands

The table below breaks down the conditional update outcomes (CU, NU, SP) across datasets and summarizes Conditional Variant Accuracy (CVA) categorized by reasoning type on ChartMuseum.

Dataset / Breakdown Proprietary Models (CU / NU / SP) Open-Source Models (CU / NU / SP) Findings & Context
ChartQA 0.929 / 0.066 / 0.005 0.851 / 0.125 / 0.023 Standard visual structures transfer robustly to counterfactual data variations.
CharXiv 0.726 / 0.214 / 0.060 0.473 / 0.378 / 0.149 Scientific multi-panel plots challenge open models, where 14.9% of updates remain stale.
ChartMuseum 0.579 / 0.327 / 0.094 0.410 / 0.432 / 0.158 Human-designed infographics trigger severe generalization failure; errors dominate updates (>58%).
ChartMuseum CVA by Type Proprietary Group Mean Open-Source Group Mean Reasoning Mechanism Bottleneck
Text 0.787 0.482 Relies on explicit textual legends and annotations; highest counterfactual generalization.
Visual / Text 0.620 0.464 Requires coordinating visual marks with nearby textual labels; intermediate robustness.
Synthesis 0.532 0.413 Demands complex aggregation and global trend extraction; significant generalization degradation.
Visual 0.460 0.331 Relies purely on visual geometry (band lengths, spatial step counting); lowest generalization.

Quality and fidelity controls confirmed the integrity of the benchmark: a systematic human audit of all 98 retained ChartMuseum charts (142 QA families) established that 100.0% preserved the task-relevant visual evidence, with 97.2% judged identically difficult to the original. Furthermore, a resolution ablation across five GPT models (reducing reconstruction resolution to 75% and 50%) lowered average RA from 0.535 to 0.513 and 0.511 respectively, matching the 0.514 OA and confirming that high baseline reconstruction accuracy reflects cleaner rendering rather than altered problem semantics.

Key Findings

  • Single-Chart Performance Masks Brittleness: High accuracy on static charts does not imply genuine visual comprehension. On ChartMuseum, proprietary models experienced an average relative decline of -20.6% on counterfactual variants (e.g., Gemini 2.5 Pro dropped from 0.732 OA to 0.529 VA, RVC -27.8%), proving that success on published benchmarks often collapses when visual evidence is updated.
  • Prevalence of Noisy Updates and Stale Predictions: Across all benchmarks, Noisy Updates (NU) formed the most frequent failure mode (up to 43.2% in open-source models on ChartMuseum), demonstrating that models frequently detect visual change but fail to construct correct visual reasoning pathways. Stale Predictions (SP) reached up to 15.8%, with blinded LLM judge evaluations showing that SP rationales exhibit 85.6โ€“89.1 claim-reuse scores with the original rationale, strongly corroborating reliance on memorized chart facts.
  • Visual Grounding as the Core Vulnerability: Generalizability drops sharply when explicit textual cues are removed. In ChartMuseum, proprietary CVA plummets from 0.787 on Text-based questions down to 0.460 on pure Visual questions (0.331 for open-source models), highlighting that modern VLMs remain heavily biased toward textual shortcuts rather than robust geometric mark parsing.

Highlights & Insights

  • Programmatic Chart Reverse Engineering: Deconstructing charts into structured data JSON and Python plotting scripts provides a scalable, generalizable foundation for synthetic multi-modal data generation that preserves domain styling while enabling unbounded data interventions.
  • Executable QA Verification: Computing target answers dynamically from altered data objects via executable Python code eliminates human re-annotation costs and removes LLM judge hallucinations, setting a rigorous standard for dynamic benchmark construction.
  • The CVA and Update Outcome Framework: Formulating evaluation through Conditional Variant Accuracy (CVA) alongside CU/NU/SP triage provides an actionable diagnostic tool to separate perceptual insensitivity, reasoning execution errors, and training-set memorization.

Limitations & Future Work

  • Coverage of Complex Chart Topologies: The pipeline excludes highly dense or non-standard chart types (~4.2% of source datasets), such as complex Sankey flow diagrams and bespoke radial plots that cannot be reliably reproduced by standard plotting libraries.
  • Fixed Layout and Schema Assumptions: The framework alters answer-relevant numerical data while holding visual layout, mark types, and prompt formulations constant; future iterations could introduce cross-schema perturbations such as converting line charts to bar charts.
  • Automated Judge Reliance: Although pairwise agreement among independent LLM judges exceeded 98%, automated answer-equivalence checking can occasionally misinterpret domain-specific numeric approximations or subtle string variations.
  • vs. ChartQA / CharXiv / ChartMuseum: Standard chart benchmarks rely exclusively on static triples where models can exploit question bias or memorized figures; CHARTOGRAPHER introduces causal data interventions to verify whether visual reasoning tracks changed evidence.
  • vs. Chart-HQA: While Chart-HQA poses hypothetical textual premises over unaltered images, CHARTOGRAPHER regenerates the chart images themselves, demanding perceptual re-grounding from raw pixels.
  • vs. ChartInsights / ChartR: Rather than testing robustness against image corruptions (noise, blur, watermarks), CHARTOGRAPHER preserves visual clarity and directly alters underlying quantitative facts, isolating high-level multimodal reasoning from low-level visual degradation.

Rating

  • Novelty: โญโญโญโญโญ Elegant reverse-engineering framework solving the scalability bottleneck of counterfactual visual data generation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 15 models, 3 major benchmarks, multi-run reproducibility tests, and blinded rationale similarity audits.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, structured definitions, and lucid case-study illustrations.
  • Value: โญโญโญโญโญ Establishes a critical benchmark paradigm to diagnose memorization and shortcut learning in modern vision-language models.