Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

Conference: ECCV2026
Paper: ECCV Paper
PDF: Conference Full Text
Dataset: AIGrounding/Diagram-MMU
Authors: Weihao Bo, Shan Zhang, Yanpeng Sun et al.
Area: Multimodal VLM
Keywords: scientific diagrams, TikZ code generation, diagram editing, object-level evaluation, agentic capability
Reading Date: 2026-09-17

Identity check: manifest ID 3381, the Chinese stub, and the cached title agree, with no subtitle change observed. The conference PDF cache contains 19 pages of main text and references, but not the cited appendices. No unverified arXiv identifier, code repository, or appendix configuration is supplied here.

TL;DR

Diagram-MMU jointly evaluates TikZ parsing, editing, and open-ended question answering across six scientific diagram domains and 12 multimodal models: Gemini-3.0 Pro reaches 86.46% question-answering accuracy but only 54.64% parsing object F1, showing that answering correctly does not imply faithful reconstruction.

Background & Motivation

Scientific diagrams are not merely images to recognize; they are structured objects that researchers need to modify. A useful assistant should convert a circuit, molecular structure, or geometry diagram into compilable TikZ, change labels, connections, or topology, and integrate the result into a LaTeX manuscript. Asking what an image depicts or which value is largest cannot establish whether the model preserves its objects, positions, and relationships.

Earlier benchmarks usually cover only part of this workflow. ChartQA and CharXiv emphasize question answering, while Plot2Code and ChartMimic focus on chart code generation. Existing TikZ benchmarks also lean toward parsing rather than unified cross-domain editing and understanding. Python and SVG outputs remain useful, but inserting a standalone rendered image is different from producing TikZ that can be compiled and edited within LaTeX. The paper therefore uses real manuals and community examples instead of generating a single diagram family for a specialist decoder.

Adding tools or intermediate steps does not necessarily resolve the problem: a model may retrieve irrelevant syntax or treat faulty intermediate code as evidence for subsequent reasoning. Core Idea: evaluate parsing, editing, and question answering on a shared scientific diagram resource, then control object context, syntax retrieval, intermediate code, and optional planning to distinguish understanding, reconstruction, and appropriate tool use.

Method

Overall Architecture

This is a benchmark rather than a new trainable architecture. Its source resources are scientific diagrams and their TikZ programs. Simplification, compilation checks, and human review precede the construction of diagram-to-code parsing (D2C-P), diagram-to-code editing (D2C-E), and diagram question answering (DQA) instances. Evaluation separately examines rendered images, code, and graphical objects, while DQA receives binary correctness judgments.

The design has four components: quality control over real diagram sources, task construction on shared sources, object and editing-partition metrics, and controlled agentic settings. The first two establish answerability and executable references; the latter two prevent visual resemblance or additional inference steps from being mistaken for reliable improvement. The 16 settings are experimental conditions, not one fixed tool-augmented algorithm.

Key Designs

1. Quality control over real diagram sources: retaining executable scientific structure

The authors collect 6,849 TikZ programs: 3,540 from official manuals and 3,309 from community resources. Manuals cover TikZ/PGF, PGFPlots, CircuiTikZ, TKZ-Euclide, ChemFig, and TikZ-Network; community sources include texample.net, TeX Stack Exchange, and GitHub tikz_favorites. This supplies authentic syntax and domain symbols from multiple packages, while limiting the evaluation to the TikZ ecosystem.

Simplification removes irrelevant preamble packages and redundant code before compilation with pdflatex, lualatex, or xelatex. Original and simplified renderings are compared by a multimodal model and then manually checked; compilation failures and changed renderings are discarded. Near-duplicates and trivial arrows, borders, or logos are removed manually. Qwen3-VL-235B-A22B proposes domain labels that humans verify. The main text reports 3,744 unique diagrams across charts, planar geometry, 3D shapes, graph structures, chemical expressions, and circuits.

2. Task construction on shared sources: separating reconstruction, modification, and understanding

D2C-P provides an image and its necessary preamble, including the document class, packages, and libraries, and requests complete executable LaTeX. The preamble reduces environment uncertainty without revealing the full diagram source; recovering the drawing program from visual objects remains the central challenge. D2C-E also starts from an image, adding an editing instruction and requesting the complete modified program. It does not supply the reference program by default.

Editing templates cover text, color, scope, and layout: label strings, element colors, local additions or transformations, and global layout or topology changes. Converting a bar chart into a line chart must preserve values and legends; changing a circuit between series and parallel configurations requires changing connectivity, not simply appearance. During construction, an agent selects two domain-relevant templates for a diagram, generates answers, checks and revises them, and passes them to human review. The paper reports cross-validation by 13 reviewers and 7,420 editing instances; selecting two templates per diagram should not be treated as an exact formula for the final sample count.

DQA uses open-ended responses rather than multiple-choice answers. Its 60 templates comprise 23 descriptive, 18 standard reasoning, and 19 what-if templates. Even descriptive questions require domain semantics: a line can represent a geometric segment or a chemical bond. Reasoning adds calculations using laws such as Ohm's law, area formulas, or graph path properties; what-if questions apply a hypothetical modification before recomputing the answer. The benchmark targets basic domain semantics and numerical computation, not expert-level scientific research ability.

3. Object and editing-partition metrics: separating resemblance, content, and successful modification

Image-level evaluation uses SSIM, CLIP Score, LPIPS, and FID; the first two are higher-is-better and the latter two lower-is-better. Code-level evaluation uses CrystalBLEU to compare generated and reference TikZ syntax and structure. Reference-code similarity must be distinguished from functional equivalence: different programs can render the same diagram, so low CrystalBLEU alone does not establish compilation failure or semantic incorrectness.

Object-level evaluation compiles generated and reference programs into DVI, converts them to SVG, and extracts a Semantic Object Model (SOM). This represents nodes, paths, text, and data series as typed elements with attributes. Type F1 compares categories such as circular and rectangular nodes; text F1 uses exact string matching for labels and numbers. Color matching uses CIEDE2000 perceptual distance with permutation-based assignment, while bounding-box matching uses an IoU threshold of 0.3.

The aggregate follows Section 3.4 of the paper:

\[ F_{1,\mathrm{avg}}=\frac{F_{1,\mathrm{type}}+F_{1,\mathrm{text}}+F_{1,\mathrm{color}}+F_{1,\mathrm{bbox}}}{4}. \]

Editing further divides code and object metrics into preserve-only and edit-only partitions, checking whether unaffected content remains intact and requested changes are implemented. Otherwise, ignoring a small edit could still produce high whole-image similarity because most of the diagram is unchanged. DQA uses Qwen3-Next-80B-A3B-Instruct to extract answers and assign 0/1 scores; accuracy is the proportion judged correct. Appendix C contains the detailed SOM extraction, per-dimension matching formulas, and judge prompt, which are absent from the available cache.

4. Controlled agentic settings: identifying which step benefits from tools or intermediate states

Table 4 defines 16 settings. S1, S6, and S12 are direct parsing, editing, and answering; the other 13 examine agentic capabilities. Context utilization supplies graphical objects as additional information, whereas parsing setting S4 asks the model to perceive objects itself before coding. These are different interventions: one provides extra information, while the other inherits the model's own perception errors.

Tool use is tested through a shared MCP interface for TikZ documentation search. The authors use Mintlify to enable selective retrieval from curated documentation, avoiding noisy general search and the cost of inserting an entire PGFPlots manual of approximately 560 pages into context. A common interface does not guarantee competent use: query content, invocation timing, and stopping behavior still depend on the model.

State-management settings require generating the original diagram code before editing or answering. Planning settings allow the model to decide whether code is needed and, in some conditions, combine it with tools or object information. Having intermediate code and deciding when to produce it thus become separate variables. The shorthand for S11 and S16 in Table 4 is not fully consistent with later descriptions; this note follows Tables 6 and 7 for the reported experiments: S16 combines optional code with tools, and S11 combines optional pre-edit code with tools. The shorthand should not be treated as an exact execution specification.

Loss & Training

The paper trains no new model and introduces no loss function. Full foundational evaluation covers 12 models, with 6 closed-source and 6 open-source models. TikZero+ 10B is a specialist baseline fine-tuned on 456K diagram-TikZ pairs; because it was not trained for editing or question answering, it is evaluated only on D2C-P. This training size describes the baseline, not the Diagram-MMU training split.

The main table uses pass@1, averaged across six diagram types. Agentic experiments use a separate subset of 300 diagrams, with 50 per domain and 1,500 instances: 300 parsing, 600 editing, and 600 question-answering instances, evaluated on 6 representative models. Generation temperatures and length limits are referenced in Appendix Table D.1, which is missing from the cache; those budgets cannot be claimed as verified.

Key Experimental Results

Main Results

The following selection comes from Table 5, the full pass@1 evaluation. Values are percentages; editing columns are object-level F1, not compilation success rates.

Model	Parsing F1avg	Editing Preserve F1avg	Editing Edit F1avg	DQA Accuracy
Gemini-3.0 Pro	54.64	65.12	40.24	86.46
Gemini-3.1 Pro	49.94	60.60	40.84	86.29
GPT-5.2	51.35	55.59	31.15	81.67
Claude-4.6 Opus	52.23	62.56	31.95	68.75
Kimi-K2.5	57.48	63.12	36.52	79.74
Qwen3-VL-8B	44.66	21.26	9.95	47.12

Kimi-K2.5 leads parsing object F1 at 57.48, but not question answering. Gemini-3.0 Pro reaches 86.46 DQA accuracy while remaining strong at editing. Different rankings across tasks provide a clearer argument that understanding differs from reconstruction than subtracting F1 from accuracy would. Specialist TikZero+ 10B reaches only 15.43 parsing F1, showing that specialist training does not automatically confer an advantage on this cross-domain distribution.

Ablation Study

These are controlled-condition comparisons from Tables 6 and 7, not module-removal ablations of a new model. Baselines and changes use the same 300-diagram subset. Resulting scores are calculated by adding each reported change to its baseline; changes are percentage points.

Source	Model and Metric	Condition Change	Subset Baseline	Change	Resulting Score
Table 6	Claude-4.6 Opus, parsing F1avg	S1 → S5, object-and-tool planning	48.5	+5.1	53.6
Table 6	Gemini-3.1 Pro, parsing F1avg	S1 → S3, syntax retrieval	47.1	-4.3	42.8
Table 6	Claude-4.6 Opus, DQA accuracy	S12 → S14, mandatory code first	80.7	+3.8	84.5
Table 6	Seed-2.0 Pro, DQA accuracy	S12 → S16, optional code plus tools	86.0	-8.8	77.2
Table 7	GPT-5.2, editing preserve F1avg	S6 → S7, supplied objects	54.9	+10.6	65.5
Table 7	GPT-5.2, editing edit F1avg	S6 → S7, supplied objects	29.3	+5.9	35.2

Key Findings

• Object context most consistently helps editing: in Table 7, all six models gain 4.1 to 10.6 points on preserve-only and 3.3 to 7.1 on edit-only F1. This supports better grounding of local edits, not improved intrinsic perception.
• Tools do not provide universal gains. Gemini-3.1 Pro loses 4.3 points with retrieval in Table 6; trajectory inspection attributes this to repeated retrieval without a stopping rule until the token limit is reached. This is trajectory-based analysis, not a separate randomized causal test.
• Optional code generation can interfere with answering. All six models decline under S15 in Table 6, but Gemini-3.1 Pro has a 0.0 change under S16; not every tool-planning condition causes degradation.
• Figure 5 shows the largest pass@k gain from 1 to 2 and saturation beyond 4. Figure 6 groups diagrams by successful parsing renders and finds corresponding editing and DQA differences. These are sampling and correlational analyses, respectively, not training gains.

Highlights & Insights

• Separating preservation from modification exposes local editing failures hidden by whole-image similarity. The principle can transfer to webpage or vector-graphics editing, provided a task-appropriate object extractor is available.
• Three tasks on shared diagram sources make error chains easier to trace. Correct domain answers alongside poor reconstruction motivate independent tests of visual grounding and program expression rather than treating QA rankings as code-generation validation.
• Controlled settings are more interpretable than one aggregate score for a complex agent. Object context, retrieval, and intermediate states can have opposing effects, helping identify where engineering changes are needed.

Limitations & Future Work

• The authors explicitly position this as a pilot agentic benchmark, not a complete evaluation of scientific writing or agency. DQA targets basic domain rules and computation, so results should not be generalized to expert scientific reasoning.
• Manual and community sources may overlap with training data; the available main text provides no verifiable contamination audit. Domain counts are also imbalanced, and macro-averaging need not reflect real-world diagram frequencies.
• The six parsing counts in Table 2 sum to 3,739, whereas the main text reports 3,744 unique diagrams. Together with 7,420 editing and 7,146 DQA instances, the task counts total 18,305. Both reported quantities are retained here; the five-diagram difference is not assigned an invented filtering explanation.
• Object matching depends on DVI/SVG extraction and thresholds; exact text matching may penalize equivalent expressions, while CrystalBLEU is not functional correctness. Structural-relation checks or execution-equivalence tests could strengthen future evaluation.
• Missing appendices prevent verification of per-model generation budgets, full prompts, extraction details, and tool failure rates. DQA also relies on one LLM judge. Reproduction should obtain the appendices and add human spot checks, confidence intervals, and cross-judge agreement tests.

Related Work & Insights

• Compared with ChartQA / CharXiv: these emphasize chart understanding, whereas Diagram-MMU jointly evaluates executable reconstruction, editing, and understanding across domains including chemistry and circuits. Its advantage is coverage, not necessarily deeper knowledge requirements.
• Compared with DeTikZify / AutomaTikZ: all use TikZ, but Diagram-MMU adds cross-task and agentic conditions. It supplies a diagnostic framework rather than a new generator guaranteed to outperform those methods.
• Compared with ChartE3 / ChartMimic: Diagram-MMU extends local/global editing and image/object evaluation ideas to six scientific domains. A promising direction is executable validation of intermediate objects or code before deciding whether they should inform later reasoning.

Rating

• Novelty: 4/5. The combination of three tasks, six domains, and agentic controls is useful; individual metrics largely build on prior work.
• Experimental Thoroughness: 4/5. Twelve models and partitioned evaluation offer breadth, but agentic tests use a small subset and reproducibility appendices are absent from this cache.
• Writing Quality: 3/5. The main argument is clear, but sample-count conventions and some setting labels need clarification.
• Value: 4/5. Useful for diagnosing scientific drawing assistants, especially reconstruction weaknesses that QA accuracy can conceal.