Skip to content

EpiCurveBench: Evaluating VLMs on Epidemic Curve Digitization

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/tberkane/EpiCurveBench
Area: Multimodal VLM
Keywords: Epidemic Curves, Chart Digitization, Vision-Language Models, Time Series Alignment, Benchmark

TL;DR

EpiCurveBench introduces a benchmark of 1,000 real-world epidemic curves paired with EpiCurveSimilarity (ECS)โ€”a dynamic-programming metric that accommodates local temporal shifts and gapsโ€”revealing that frontier VLMs reach only 52.3% ECS on dense, real-world chart digitization.

Background & Motivation

Vision-language models (VLMs) have shown remarkable promise in extracting structured numerical tables from chart images, but the benchmarks and evaluation metrics tracking progress in this area have reached an impasse. Existing public benchmarks such as ChartQA are dominated by simple, predominantly synthetic infographics featuring sparse data points and explicitly printed numbers; frontier models now surpass 89% Relative Mapping Similarity (RMS) on these suites, leaving little headroom to discriminate between capable architectures. Concurrently, dominant evaluation metrics like RMS and SCRM conceptualize extracted chart tables as sets of unordered key-value pairs, where any slight deviation in text labels or minor temporal offset is penalized as a complete mismatch, erroneously conflating value-accurate extractions shifted by a single day with catastrophic hallucinations.

This evaluation deficiency is especially acute in high-stakes public health contexts. Epidemic curves ("epicurves") represent the foundational visualization format through which public health agencies communicate outbreak dynamics, including case incidences, hospitalizations, and mortality trajectories. Millions of outbreak records spanning centuries remain trapped solely as raster images inside legacy PDF surveillance bulletins; unlocking this data via automated digitization would directly provision training data for forecasting models that inform epidemic containment. However, real-world epicurves are profoundly challenging: they contain dense temporal points (median 156 points, scaling up to 8,592 points per chart), overlapping or stacked multi-series, low-contrast historical printings, rotated text labels, and complex annotations, requiring up to three hours for a human expert to digitize a single intricate figure.

To bridge the gap between artificial benchmarks and dense, structured real-world time-series extraction, this work establishes an empirical stress-test across diverse global outbreaks and designs an order-aware alignment metric reflecting practical downstream utility. Core idea: construct EpiCurveBench, a 1,000-chart benchmark of real-world epidemic curves spanning 176 years, and formulate EpiCurveSimilarity (ECS), an alignment metric based on Edit Distance with Real Penalty that tolerates minor temporal shifts while penalizing data gaps proportionally to reflect downstream epidemiological validity.

Method

Overall Architecture

The EpiCurveBench and ECS evaluation pipeline comprises three core phases: curation of a diverse, dual-source empirical benchmark with multi-stage quality verification; formulation of the temporally aware ECS metric using dynamic programming with scale-normalized real penalties; and systematic evaluation of diverse general-purpose and specialized VLMs across varying reasoning budgets and code-execution configurations. The benchmark collects 1,000 openly licensed real-world epidemic charts divided into a source-aligned tabular subset and an extensively audited, visually heterogeneous manual subset. During evaluation, models extract structured tabular data via zero-shot prompts; ECS pairs extracted and reference series via label similarity, computes an optimal edit alignment over time indices, and evaluates alignment fidelity against downstream epidemiological parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Raw Public Health Reports & Images<br/>13 source types ยท 14 diseases ยท 1849โ€“2025"] --> B["Dual-Source Benchmark Construction<br/>Set 1 tabular audit + Set 2 three-stage DINOv2 de-duplication"]
    B --> C["VLM Zero-Shot Extraction to TSV<br/>Uniform prompt requesting gapless time series"]
    C --> D["Temporally Aware Metric ECS<br/>NLS series matching + ERP dynamic programming + range-normalized penalties"]
    D --> E["Downstream Utility Validation<br/>Spearman correlation against total counts / peak timing / peak magnitude / growth rate"]

Key Designs

1. Dual-Source Benchmark Construction: Complementary Tabular Alignment and Three-Stage Visual De-Duplication

To eliminate the artificial simplicity of synthetic infographics, EpiCurveBench curates 1,000 authentic epicurve images across two complementary cohorts. Set 1 consists of 900 epicurves paired with original machine-readable tabular datasets released by reporting organizations (predominantly the US CDC alongside international health departments). An audit of a random 20% sample (180 charts) verified 100% alignment between raster figures and source tables, establishing a scalable, verified ground truth. Set 2 contributes 100 manually annotated epicurves selected from an initial pool of 1,397 candidates to maximize stylistic, geographical, and temporal variety across 14 diseases and 36 countries. To prevent visual redundancy, candidate images were filtered through a three-stage curation protocol: initial manual selection for visual complexity; embedding with DINOv2 (ViT-L/14) and DBSCAN clustering on cosine distance to eliminate near-duplicates (similarity > 0.9, replacing 8 redundant charts); and independent dual-annotator calibration via WebPlotDigitizer. Across Set 2, annotators achieved a mean inter-annotator ECS of 90.9% (median 95.4%), with discrepancies confined exclusively to ambiguous boundary data points rather than series confusion.

2. Temporally Aware Metric ECS: Edit Distance with Real Penalty and Range-Normalized Cost

Standard chart extraction metrics (RMS, SCRM) rely on exact string matching of composite keys (series name + timestamp label), meaning a one-day global offset reduces the extraction score from near-perfect to zero. ECS solves this by adapting Edit Distance with Real Penalty (ERP). Given extracted series \(E=\{e_1, \dots, e_m\}\) and ground-truth series \(T=\{t_1, \dots, t_n\}\), series pairing is conducted via Normalized Levenshtein Similarity (NLS) with a matching threshold of 0.5. For each matched pair \(p=(p_1, \dots, p_M)\) and \(t=(t_1, \dots, t_N)\), the optimal alignment cost \(C(M, N)\) is computed through dynamic programming:

\[C(i, j) = \min \begin{cases} C(i-1, j-1) + D_\theta(p_i, t_j) \\ C(i-1, j) + \lambda \\ C(i, j-1) + \lambda \end{cases}\]

with boundary conditions \(C(0,0)=0\), \(C(i,0)=i\cdot\lambda\), and \(C(0,j)=j\cdot\lambda\), setting gap penalty \(\lambda=1.0\). The substitution cost \(D_\theta(p_i, t_j)\) measures the relative value discrepancy \(\delta_{ij} = |p_i - t_j| / (y_{\max} - y_{\min})\), normalized by the ground-truth series' vertical span, and clipped to 1.0 when exceeding tolerance \(\theta=0.01\) (1% of the y-axis range). The final series score maps alignment cost to \([0, 1]\):

\[\text{ECS}(p, t) = 1 - \frac{C(M, N)}{\#\text{matches} + \#\text{gaps}}\]

Unmatched reference series receive a score of zero, and image-level scores average all reference series. This formulation grants tolerance to minor local time shifts and axis-label formatting idiosyncrasies while penalizing omitted or false data points strictly in proportion to missing steps.

3. Downstream Utility Validation: Grounding Extraction Metrics in Epidemiological Modeling Validity

A robust evaluation metric must correlate faithfully with the accuracy of downstream scientific analytics derived from extracted data. While Dynamic Time Warping (DTW) allows non-linear monotonic alignment, it lacks an explicit penalty for unmatched coordinates; consequently, a severely truncated extraction can warp across the entire ground truth and receive an artificially high score. To confirm that the explicit gap penalty in ECS preserves scientific utility, the authors evaluate four standard epidemiological summary statistics: aggregate disease burden (Total-count relative error), epidemic crest timing (Peak timing index difference), healthcare surge load (Peak magnitude relative error), and transmission velocity (early ascending-phase log-incidence Growth rate correlation). On thousands of matched series across all models, ECS demonstrates strong, statistically significant rank correlations with lower errors in all four epidemiological parameters, confirming that penalizing unextracted points is essential to preventing downstream model distortion.

Key Experimental Results

Main Results

The benchmark evaluates nine distinct methods spanning closed frontier models, open-weight multimodal architectures, and specialized chart systems. General-purpose VLMs were assessed under Minimal reasoning, High reasoning, and High reasoning combined with a Python code-execution environment capable of dynamic zooming and cropping.

Model Reasoning / Tool Config Full ECS (%) Set 1 ECS (%) Set 2 ECS (%) Inference Cost ($)
OneChart Default 9.4 8.3 18.5 โ€”
TinyChart Default 11.4 10.7 18.4 โ€”
Qwen3-VL Minimal 26.1 25.2 35.2 0.36
Qwen3-VL High 27.5 26.7 34.7 1.98
Qwen3-VL High+Code 29.5 28.6 37.3 4.82
GPT-5.2 Minimal 45.4 45.1 46.6 1.71
GPT-5.2 High 43.9 43.3 50.0 16.96
GPT-5.2 High+Code 44.8 44.9 44.1 19.64
Claude Opus 4.5 Minimal 41.3 41.6 39.4 3.92
Claude Opus 4.5 High 42.0 42.1 41.4 29.60
Claude Opus 4.5 High+Code 40.5 40.5 40.2 82.41
Gemini 2.5 Pro Minimal 40.5 39.4 50.2 6.29
Gemini 2.5 Pro High 52.3 51.8 56.9 15.75
Gemini 2.5 Pro High+Code 48.0 45.2 69.4 19.16
Xiaomi MiMo-VL 2.5 High 50.4 50.2 51.9 2.32
MiniMax-M3 High 47.0 46.4 53.3 8.09
Llama-4-Maverick High 26.8 26.3 31.4 1.15

Metric Discrimination and Downstream Correlation

To validate the diagnostic superiority of ECS over key-value metrics (RMS, SCRM) and alignment metrics without gap penalties (DTW), the study examines model score dispersion (Spread) and Spearman rank correlations (\(r\)) against downstream epidemiological summary errors.

Model / Metric RMS (%) SCRM (%) DTW (%) ERP (%) ECS (%)
OneChart 4.4 4.3 26.4 7.5 9.4
TinyChart 8.5 8.5 35.6 9.3 11.4
Qwen3-VL (High) 18.0 17.9 52.2 27.5 27.5
GPT-5.2 (High) 18.6 18.6 65.8 38.3 43.9
Claude Opus 4.5 (High) 22.8 23.0 62.4 38.5 42.0
Gemini 2.5 Pro (High) 20.5 20.8 72.9 45.3 52.3
4-VLM Spread (Max - Min) 4.8 5.2 20.7 17.9 24.8

Correlation analysis across matched (series, model) pairs illustrates the necessity of penalizing unextracted points:

Downstream Summary Statistic \(r_{\text{ECS}}\) \(r_{\text{TWED}}\) \(r_{\text{DTW}}\) \(r_{\text{Soft-DTW}}\) \(r_{\text{LCSS}}\) Series Count \(n\)
Total-count error -0.58 -0.62 -0.38 -0.31 +0.11 944
Peak timing error -0.51 -0.53 -0.15 -0.11 +0.12 10,512
Peak magnitude error -0.29 -0.25 -0.08 -0.08 -0.24 7,484
Growth-rate fidelity +0.19 +0.20 +0.08 +0.06 +0.06 4,862
**Mean Absolute Correlation ($ r $)** 0.39 0.40 0.18 0.14

Key Findings

  • Real-World Extraction Is Far from Saturated: The top-performing system (Gemini 2.5 Pro at high reasoning) reaches only 52.3% ECS on EpiCurveBench, demonstrating that real-world chart digitization poses unresolved perceptual bottlenecks. Specialized chart models (OneChart at 9.4%, TinyChart at 11.4%) fail substantially due to synthetic-to-real distribution shifts and limited parameter capacity.
  • Open-Weight Models Rival Frontier Systems: Xiaomi MiMo-VL 2.5 achieves 50.4% ECS, placing second overall within 1.9 points of Gemini 2.5 Pro while operating at approximately \(6.8\times\) lower inference cost ($2.32 vs. $15.75). MiniMax-M3 achieves 47.0%, surpassing both GPT-5.2 and Claude Opus 4.5.
  • Reasoning Budgets and Tool Access Exhibit Mixed Utility: Escalating reasoning effort yields large gains for Gemini 2.5 Pro (+11.8 ECS) but delivers modest returns for Qwen3-VL (+1.4) and slightly degrades GPT-5.2 (-1.5). Augmenting models with Python code execution for cropping hurts Claude (-1.5) and Gemini (-4.3) while dramatically inflating costs (Claude High+Code reaches $82.41), primarily due to imprecise visual bounding boxes that cause fragmented, duplicate, or missing temporal segments during stitching.
  • Key-Value Metrics Compress Performance and Misrank Leaders: RMS and SCRM compress all four general-purpose VLMs into a tiny 4.8โ€“5.2 point band and falsely identify Claude as the top model. Conversely, ECS expands the models across a 24.8-point spread and correctly identifies Gemini as superior. Metrics incorporating explicit gap penalties (ECS, TWED) correlate 1.5โ€“3.6x more strongly with downstream epidemiological errors than gap-free warping measures (DTW, Soft-DTW, LCSS).

Highlights & Insights

  • From Rigid Lexical Matching to Order-Preserving Elastic Alignment: Demonstrates that chart digitization must not be framed as unordered dictionary extraction; adapting dynamic-programming sequence alignment grants natural robustness against single-day shifts while faithfully tracking curve morphology.
  • Isolating Series Truncation as the Primary Mode of Failure: Diagnostic error decomposition shows that on dense, long-horizon charts, missed data points due to premature model truncation account for the largest proportion of accuracy loss, rather than minor coordinate noise.
  • Domain-Agnostic Metric Generalizability: Although motivated by outbreak surveillance, ECS assumes only ordered, time-indexed numerical series with sparse tick marks. It can directly serve as an evaluation standard for extracting dense sensor readings, financial tick data, and environmental monitoring graphs.

Limitations & Future Work

  • Geographic and Institutional Skew in Ground Truth: Set 1 predominantly originates from the US CDC due to the scarcity of agencies releasing raw tabular data alongside outbreak bulletins. Although complexity-matched controls verify that the CDC performance gap is driven by series length rather than visual memorization, expanding open-access surveillance figures from the Global South remains critical.
  • Disease Class and Formatting Asymmetry: The dataset is heavily weighted toward respiratory pathogens (Influenza, COVID-19, and RSV represent 88% of charts) and non-cumulative line charts (71.4%). Expanding coverage to vector-borne outbreaks, cumulative tracking, and diverse multi-axis bar charts is necessary.
  • VLM Spatial Localization Deficits in Image Sub-Cropping: Code-assisted zooming degraded overall accuracy due to inaccurate sub-region cropping, highlighting the need for improved spatial grounding and coordinate tracking in visual reasoning backbones.
  • vs ChartQA / PlotQA: Prior benchmarks feature predominantly synthetic infographics with simple layouts and sparse numbers, where frontier VLMs exceed 89% RMS. EpiCurveBench introduces real-world historical and modern outbreak curves with high density (up to 8,592 points), dropping frontier performance to 18โ€“23% RMS and 52.3% ECS.
  • vs RMS / SCRM: Existing chart metrics evaluate set overlaps of composite text keys and zero-out predictions with minor date-format discrepancies. ECS evaluates the underlying numerical trajectory via dynamic programming with scale-normalized tolerances.
  • vs Dynamic Time Warping (DTW): DTW permits unconstrained many-to-one warping without penalizing missing points, enabling truncated outputs to claim deceptively high scores. ECS incorporates an explicit gap penalty, achieving 1.5โ€“3.6x stronger correlation with real epidemiological summary metrics.

Rating

  • Novelty: โญโญโญโญ [Introduces an order-aware alignment metric with proportional gap penalties tailored for continuous chart digitization]
  • Experimental Thoroughness: โญโญโญโญโญ [Evaluates 9 models across multiple inference modes, conducts comprehensive hyperparameter sensitivity sweeps, error breakdowns, and rigorous downstream validation]
  • Writing Quality: โญโญโญโญโญ [Clear motivation, mathematically rigorous metric definition, and crisp presentation of complex error breakdowns]
  • Value: โญโญโญโญโญ [Directly unlocks decades of historical outbreak surveillance figures trapped in static documents for epidemic forecasting and computational health]