ChartREG++: Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Chart Referring Expression Grounding, Multimodal Large Language Models, Chart Instance Segmentation, Set-of-Mark Prompting, Fine-Grained Chart Understanding
TL;DR¶
To tackle the limitations of existing chart grounding benchmarks constrained by coarse bounding boxes, single targets, and narrow textual cues, this paper introduces ChartREG++ supporting point/box/mask formats, multi-target grounding, and 13 fine-grained referring cue types, alongside an LLM-free Matplotlib Artist-hierarchy mask synthesis pipeline that substantially enhances chart referring expression segmentation.
Background & Motivation¶
With the rapid emergence of multimodal foundation models across chart question answering, structured parsing, and chart-to-text summarization, automated chart understanding is transitioning from academic exploration to practical deployment. Nevertheless, early systems almost universally produce only textual conclusions or descriptions without explicitly linking answers back to visual evidence within the source chart. This architectural limitation severely hampers direct verification by human users, making it difficult to trust automated conclusions in high-stakes domains such as scientific literature and financial reporting. To cultivate verifiable and trustworthy multimodal chart intelligence, incorporating fine-grained visual grounding signals into reasoning traces and evaluating evidence attribution have become crucial research frontiers.
While referring expression grounding serves as a canonical diagnostic benchmark for spatial localization and vision-language alignment in natural images, porting it to the chart domain exposes four severe bottlenecks in existing benchmarks. First, prior benchmarks predominantly represent spatial evidence using coarse, axis-aligned bounding boxes, which are ill-suited for slender, irregular, or overlapping chart primitives such as error bars and polylines. Second, existing formulations overwhelmingly assume only one or two target instances (over 80% to 90% of samples in RefChartQA and ChartLens), leaving complex multi-instance references severely under-explored. Third, language expressions over-rely on textual anchors or simplistic extreme-value data rankings, failing to capture diverse perceptual dimensions such as shapes, line styles, fill patterns, and cross-series relationships. Fourth, the coverage of chart element types remains minimal, rarely supporting hierarchical representations from atomic primitives to composite semantic components.
To resolve these interconnected challenges, this work introduces a synergistic solution covering both benchmark evaluation and high-quality synthetic supervision. The authors construct ChartREG++, a comprehensive benchmark derived from realistic chart sources (ChartMimic and ECDBench), spanning 18 multi-granularity chart element types, an average of 9.7 referred targets per query, and complete multi-format localization support (points, bounding boxes, and pixel-accurate instance masks). Furthermore, addressing the prohibitive annotation cost of dense chart instance masks, the authors exploit the intrinsic alignment between Matplotlib's internal Artist hierarchy and rendered screen primitives to establish a fully automated, LLM-free mask synthesis pipeline. Core idea: exploit the deterministic correspondence between Matplotlib's internal Artist hierarchy and rendered primitives to construct an LLM-free, multi-granularity instance mask synthesis pipeline, and establish a two-stage "detect-then-select" chart referring segmentation framework.
Method¶
Overall Architecture¶
The methodology consists of two core contributions: the construction of the multi-format, multi-target, multi-cue ChartREG++ benchmark, and an LLM-free instance mask synthesis pipeline that powers a two-stage chart referring segmentation framework.
In the benchmark construction pipeline, the taxonomy establishes 18 chart element types and 3 primary referring cue categories spanning 13 subtypes. Candidate queries are generated by prompting state-of-the-art LLMs (Gemini, GPT, DeepSeek) conditioned on executable code, followed by two-round rigorous human verification ensuring precision, uniqueness, and target correctness. In the grounding system, a two-stage "detect-then-select" pipeline is adopted: Stage 1 deploys a Mask2Former model fine-tuned on the synthesized multi-granularity chart masks to produce high-recall candidate instance proposals; Stage 2 leverages Set-of-Mark (SoM) prompting to overlay candidate masks with alphanumeric identifiers, enabling a multimodal large language model (MLLM) to select the subset of marks that satisfy the complex multimodal referring expression.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input chart image I and referring expression q"] --> B["Matplotlib Internal Artist Hierarchy Mapping<br/>Trace executable plotting APIs to rendered graphic primitives"]
B --> C["LLM-Free Multi-Granularity Mask & Label Synthesis<br/>Primitive/part/composite semantic level extraction"]
C --> D["Mask2Former Chart Candidate Mask Generator<br/>Fine-tuned on synthetic chart masks for high-recall proposals"]
D --> E["Set-of-Mark Prompting & Multimodal Alignment<br/>MLLM selects matching mask IDs under diverse referring clues"]
E --> F["Output Target Grounding (Points / BBoxes / Masks)"]
Key Designs¶
1. Matplotlib Internal Artist Hierarchy Mapping: Code-to-Pixel Deterministic Binding Conventional chart vector parsing relies on SVG paths and groups, which lack high-level semantic abstractions, produce brittle rule-based heuristic aggregations, and fail to distinguish visually indistinguishable primitives (such as standalone scatter points versus line markers). To overcome this fundamental limitation, this work directly hooks into the Python execution runtime to trace the Matplotlib Artist object hierarchy returned by plotting APIs (such as Line2D objects from ax.plot or Patch collections from ax.boxplot). By extracting the exact canvas coordinates and rendered pixels contributed by each Artist during rasterization, the pipeline creates a clean, deterministic mapping linking code-level visual objects to pixel-accurate image regions.
2. LLM-Free Multi-Granularity Mask and Label Synthesis: Multi-Level Semantic Disambiguation Extracting masks purely from visual appearance frequently leads to semantic ambiguity among identical geometric primitives, whereas prompting LLMs to infer data points induces hallucinations and scales poorly. The proposed mechanism combines internal Artist metadata with API calling contexts to automatically generate multi-granularity masks and disambiguated fine-grained labels. For instance, for box plots, the system simultaneously extracts holistic semantic units ("Full Box" including whiskers, caps, median line, and box patch) and decomposes them into atomic constituents ("BoxMedianLine", "BoxPlot_Boxpatch", error bars). Markers plotted via ax.plot are distinguished from points created via ax.scatter, producing clean, hierarchical instance supervision at scale without requiring any human annotation or LLM inference.
3. ChartREG++ Multi-Dimensional Referring Cue Taxonomy: Systematic Multi-Target Disentanglement Existing chart grounding benchmarks are dominated by single-target queries and simple extreme-value cues. ChartREG++ formalizes a comprehensive taxonomy comprising 3 primary categories and 13 sub-dimensions: Data Features (Value-Range Filtering, Rank-Band Set Selection, Local-Structure Patterns, Cross-Series Relations), Visual Features (Color Attributes, Shape Style, Line/Stroke Style, Fill Style), and Textual/Localization Features (Axis labels, Non-data axis tick values/positions, Legend entries/positions, Subplot titles/positions, Direct on-chart annotations), along with hybrid feature compositions. Queries are constrained to start with explicit referent subjects and enforced to strictly depend on the designated clue category, enabling rigorous diagnostic profiling over an average of 9.7 targets per query.
4. Two-Stage Chart Referring Segmentation Framework: Decoupling Proposal from Alignment End-to-end segmentation models (e.g., Sa2VA) struggle to separate individual dense chart instances and frequently drop thin-line elements, while general-purpose segmentation foundations like SAM3 generate excessive background proposals and composite fragments. This work reformulates the task into candidate generation followed by discrete mark selection. By deploying a Mask2Former candidate generator trained on high-quality chart synthetic masks, candidate quality for thin lines and fine chart elements is dramatically boosted. Set-of-Mark prompting then projects numerical tags onto the candidates, allowing the MLLM to perform logical and visual reasoning over discrete symbolic tokens rather than directly predicting continuous spatial coordinates.
Loss & Training¶
The candidate proposal network adopts the standard Mask2Former architecture trained on 40K chart samples from ChartCoder and the ECDBench training split. The model is optimized using standard binary cross-entropy loss, Dice loss, and classification cross-entropy loss: $\(\mathcal{L} = \lambda_{\text{cls}} \mathcal{L}_{\text{cls}} + \lambda_{\text{bce}} \mathcal{L}_{\text{bce}} + \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}\)$ In the second stage, downstream MLLMs (Qwen3-VL, InternVL3.5) perform zero-shot Set-of-Mark prediction via greedy decoding without any task-specific parameter fine-tuning.
Key Experimental Results¶
Main Results¶
The authors evaluate state-of-the-art multimodal foundation models and visual grounding systems on ChartREG++. Table 1 details point- and bounding box-based grounding metrics (macro Precision, Recall, and F1), while Table 2 summarizes instance segmentation grounding performance.
| Setting & Model Backbone | ECDBench P (%) | ECDBench R (%) | ECDBench F1 (%) | ChartMimic P (%) | ChartMimic R (%) | ChartMimic F1 (%) |
|---|---|---|---|---|---|---|
| Point-Based Grounding | ||||||
| Gemini3-Pro | 32.9 | 39.4 | 33.4 | 36.8 | 44.8 | 37.0 |
| GPT-5.2 | 18.3 | 24.2 | 18.7 | 31.7 | 40.3 | 30.5 |
| Molmo-v2-7B | 31.0 | 33.6 | 30.0 | 46.9 | 51.7 | 45.2 |
| Qwen3-VL-8B | 8.4 | 10.6 | 8.5 | 10.5 | 13.4 | 10.3 |
| BBox-Based Grounding | ||||||
| Gemini3-Pro | 24.2 | 23.9 | 23.9 | 30.3 | 29.8 | 29.8 |
| GPT-5.2 | 10.5 | 10.4 | 10.4 | 23.3 | 23.0 | 23.0 |
| Qwen3-VL-7B | 8.6 | 7.9 | 8.1 | 15.6 | 14.9 | 14.9 |
| Qwen3-VL-32B | 15.0 | 14.8 | 14.6 | 15.8 | 15.7 | 15.5 |
| InternVL3.5-8B | 2.2 | 2.0 | 2.0 | 5.8 | 5.4 | 5.5 |
For segmentation grounding, the two-stage framework trained on synthetic masks is evaluated against zero-shot E2E models and zero-shot SAM3 proposals:
| Grounding Paradigm & Candidates | ECDBench P (%) | ECDBench R (%) | ECDBench F1 (%) | ChartMimic P (%) | ChartMimic R (%) | ChartMimic F1 (%) | mIoU_union (%) |
|---|---|---|---|---|---|---|---|
| Sa2VA-Qwen2.5-VL-7B (E2E) | โ | โ | โ | โ | โ | โ | 21.00 |
| Sa2VA-InternVL3-14B (E2E) | โ | โ | โ | โ | โ | โ | 24.00 |
| SAM3-SoM-QwenVL-32B | 28.18 | 24.22 | 25.01 | 29.55 | 21.55 | 22.91 | 22.13 |
| SAM3-SoM-InternVL3.5-30B-A3B | 24.57 | 21.44 | 21.84 | 24.05 | 18.39 | 19.16 | 18.79 |
| Ours-SoM-QwenVL-32B (All Segs) | 44.64 | 52.55 | 43.82 | 43.75 | 49.85 | 43.79 | 42.93 |
| Ours-SoM-InternVL3.5-30B-A3B | 34.60 | 35.58 | 34.25 | 41.81 | 40.88 | 37.29 | 34.70 |
| Ours-SoM-QwenVL-32B (Gold Category) | 59.65 | 57.46 | 56.77 | 59.78 | 58.34 | 56.73 | 52.42 |
| Ours-SoM-QwenVL-8B (Oracle GT Segs) | 75.85 | 66.85 | 67.86 | 76.35 | 68.14 | 68.38 | 64.26 |
Out-of-distribution transfer to authentic non-Matplotlib charts (ChartLens-ChartQA subset) using Qwen3VL-32B is summarized below:
| Candidate Proposal Method | HBar F1 (%) | VBar F1 (%) | Pie F1 (%) | Line+Marker Det. Rate (%) | Avg. False Mask Area (%) |
|---|---|---|---|---|---|
| ChartLens-SOM | 78.79 | 92.86 | 87.50 | 22.55 | 40.43 |
| SAM3-SOM | 36.36 | 78.90 | 84.97 | 3.92 | 0.02 |
| Ours-SOM | 90.91 | 88.74 | 94.52 | 75.49 | 1.43 |
Ablation Study¶
To isolate the genuine utility of the synthesized mask supervision from architectural variations, a controlled experiment was conducted using the identical Mask2Former architecture, initial weights, training schedule, and evaluation protocol on ECDBench:
| Training Supervision Source | \(AP_{[0.50:0.95]}\) | \(AP_{50}\) | \(AP_{75}\) | Core Observation |
|---|---|---|---|---|
| Bbox-as-mask | 0.164 | 0.313 | 0.148 | Coarse bounding rectangles severely degrade thin element boundaries |
| Bbox-prompt-SAM3-as-mask | 0.280 | 0.410 | 0.293 | Bbox-prompted SAM3 pseudo masks partially recover shapes but lack precision |
| Ours (Pixel-Level Synthetic Masks) | 0.313 | 0.462 | 0.315 | Direct Artist-derived masks deliver superior boundary precision |
In standard COCO-style instance segmentation (Table 9), Ours-Detection achieves 51.59% mAP and 69.34% \(mAP_{50}\), decisively outperforming off-the-shelf GroundingSAM2 (0.58% mAP) and SAM3-Detection (10.25% mAP).
Key Findings¶
- Existing models struggle with chart grounding: Even top commercial models (Gemini3-Pro, Molmo-v2-7B) achieve F1 scores below 50% across point and bounding box settings, with box-based F1 substantially trailing point-based F1 due to tighter geometric constraints.
- Mask supervision quality is the decisive bottleneck: Controlled ablations reveal that training Mask2Former with tight-box-prompted SAM3 pseudo masks achieves only 0.410 \(AP_{50}\), trailing direct Artist-synthesized masks (0.462 \(AP_{50}\)) by over 5 points, confirming that generic vision models cannot accurately reconstruct thin chart elements from boxes alone.
- Data-reasoning clues exhibit steep performance drops: Breakdown analysis demonstrates strong performance on perceptual visual features (F1: 0.54-0.66) and textual localization cues (F1: 0.58-0.68), but a sharp drop on data features requiring relational ranking and arithmetic comparison (F1: 0.33-0.49).
- Scale and code complexity degrade grounding: Performance steadily deteriorates as chart code length exceeds 100 lines and target instance count surpasses 25, validating that dense visual clutter and multi-instance reasoning are critical failure modes for current MLLMs.
- Significant headroom remains even under oracle proposals: When provided with perfect ground-truth candidate masks (Oracle setting), QwenVL-8B achieves 67.86% F1, demonstrating that linguistic alignment over complex referring logic leaves immense room for future research.
Highlights & Insights¶
- Zero-cost, high-fidelity supervision via runtime Artist tracing: Rather than relying on fragile SVG scraping or hallucination-prone LLM code translation, directly tapping into Python's Matplotlib Artist hierarchy extracts pixel-exact instance masks across multi-granularity hierarchies at zero manual annotation cost.
- Multimodal grounding benchmark expansion: Transitions chart grounding from primitive single-target bounding boxes to a rich diagnostic testbed featuring multi-target sets (average 9.7 targets), point/box/mask formats, and 13 fine-grained referring cue types.
- Robust cross-engine generalization: Models trained exclusively on Matplotlib-synthesized masks generalize seamlessly to non-Matplotlib charts (ChartLens-ChartQA), lifting line marker detection rates from 22.55% to 75.49% while drastically curbing false positive area coverage.
Limitations & Future Work¶
- Reliance on Matplotlib runtime environment: The mask extraction pipeline currently depends on the object model of Matplotlib; extending this mechanism to other visualization ecosystems like D3.js, Vega-Lite, and ECharts represents an essential next step.
- Absence of aligned multimodal training triplets: The synthesized data currently provides image-mask pairs rather than full (image, expression, target mask) triplets required for end-to-end referring segmentation models like Sa2VA.
- Homogeneous target assumption: The current formulation assumes referred targets within a single expression belong to the same semantic element category, leaving heterogeneous composite queries (e.g., "the tallest bar and its associated error bar and value label") for future investigations.
Related Work & Insights¶
- vs ChartRef (Tjandrasuwita et al., 2025): ChartRef targets single-target bounding box localization relying predominantly on textual cues and LLM-assisted data inference; ChartREG++ provides multi-target (9.7 targets average), multi-format (point/box/mask), and 13 fine-grained multimodal cue evaluations, powered by an LLM-free mask synthesis pipeline.
- vs RefChartQA (Vogel et al., 2025) & ChartLens (Suri et al., 2025): These attribution QA benchmarks require text answer generation where over 90% of bounding box evidence is constrained to single/two targets and basic rank cues; ChartREG++ isolates referring localization and evaluates structural, stylistic, and relational cues across 18 element types.
- vs InfoDet (Zhu et al., 2026): InfoDet parses SVG elements for coarse detection boxes lacking hierarchical semantic relations; ChartREG++ preserves multi-granularity primitive-part-composition structures directly through Python's object hierarchy.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering multi-target, multi-format chart referring expression benchmark with an ingenious LLM-free Artist-level mask synthesis pipeline.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluations covering closed-source models, open-source MLLMs, controlled supervision ablations, and cross-engine real-world transfer.
- Writing Quality: โญโญโญโญโญ Clear exposition, disciplined mathematical and linguistic formulations, and exhaustive supplementary analyses.
- Value: โญโญโญโญโญ Establishes a foundational standard and training paradigm for verifiable and interpretable multimodal chart understanding.