Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility

Conference: ECCV2026
Paper: ECCV Paper
Authors: Honglin Lin, Chonghan Qin, Zheng Liu, Qizhi Pei, Yu Li, Zhanping Zhong, Xin Gao, Wei Li, Wentao Zhang, Yanfeng Wang, Conghui He, Lijun Wu
Area: Image Generation
Keywords: scientific image synthesis, programmatic generation, visually necessary questions, structural precision, multimodal reasoning

TL;DR

SciGenBench measures whether scientific images preserve usable reasoning information, comparing direct pixel generation with an understand-plan-code ImgCoder workflow; Gemini-3-Pro-ImgCoder achieves a 77.87% image-level inverse validation rate under the default evaluator, and synthetic data also improves downstream multimodal reasoning.

Background & Motivation

A scientific illustration must do more than turn text into a plausible picture: circuit connectivity, function extrema, and molecular valence can determine the answer to a problem. Texture quality and broad semantic similarity, strengths of natural-image generation, do not guarantee these discrete relationships or quantitative constraints. In synthetic training data, an attractive but incorrect diagram becomes erroneous supervision rather than harmless decoration.

Prior scientific visualization research includes code generation, template rendering, and research-figure synthesis, but presentation quality and reasoning utility remain different objectives. This paper therefore compares pixel-based and programmatic generation, constructs an information-oriented evaluation, and checks whether training on the resulting images improves problem solving. Its main contribution is a unified study protocol and utility evidence, not a new diffusion architecture or a formal solver for all scientific reasoning.

Core Idea: define scientific image quality through reliable recovery of structure and information, use executable code to control diagram constraints explicitly, and connect image-level validation to downstream reasoning training.

Method

Overall Architecture

Inputs are filtered scientific problem descriptions; outputs include generated images, evaluation results, and image-text problems for training. The workflow constructs scientific instructions, compares two synthesis paradigms, checks images with visually necessary questions and multidimensional judging, and then runs downstream reinforcement learning experiments. SciGenBench contains approximately 1.4K problems across 5 subjects and 25 image types.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Scientific text and real references"] --> Data["Scientific Instruction Construction"]
    Data --> Gen["Two-Paradigm Synthesis<br/>Pixel generation or ImgCoder"]
    Gen --> Eval["Visual Necessity and<br/>Hybrid Evaluation"]
    Data -->|Source instructions supply atomic facts| Eval
    Eval --> Train["Downstream Utility Testing"]
    Train --> Output["Image quality and reasoning performance"]

Key Designs

1. Scientific Instruction Construction: identify which text should become an image

MegaScience and WebInstruct-verified provide the main problem sources. Instead of visualizing every scientific passage, the pipeline removes abstract derivations and other content without concrete visual structure. Gemini-3-Flash jointly filters for visualizability and performs fine-grained classification in one inference pass, retaining descriptions that correspond to inspectable geometric entities, physical apparatus, or other diagram structures.

The taxonomy has two levels, subject and image type: Mathematics, Physics, Chemistry, Biology, and Universal for cross-domain structures. Image types include molecular structures, circuit diagrams, charts, and experimental setups. This organization helps distinguish scientific-knowledge failures from difficulties with dense labels or topology, rather than judging methods only by subject-level averages.

A separate reference source is the vision-optional subset of SeePhys, named SciGenBench-SeePhys. Its real images participate in the evaluations and anchor comparisons between real and synthetic imagery. Only this subset supports reference-based conventional metrics; their evaluation scope must not be described as the entire SciGenBench dataset.

2. Two-Paradigm Synthesis: separate structural planning from final rendering

The pixel-based route translates text directly into images, but its prompts still impose three constraints: visualize every entity, avoid leaking calculations or final answers, and use a clear textbook style. The comparison is therefore against a science-adapted prompting workflow rather than an arbitrary short prompt. Evaluated models include Qwen-Image, HunyuanImage-3.0, GPT-Image-1.5, and Nanobanana-Pro.

ImgCoder first understands the problem, then plans, and finally emits executable drawing code for deterministic rendering. Planning specifies four aspects: image content lists entities and relationships; layout establishes coordinates and topology to avoid unintended overlap; labels specify text and anchor points; drawing constraints check domain laws while prohibiting answer leakage. Relationships otherwise implicit in pixels become executable structural specifications. Coordinates and curves can be computed numerically instead of merely looking approximately right.

This remains a prompt-driven programmatic baseline, not a newly trained drawing network, and code is not inherently scientifically valid. Incorrect understanding, planning, or formulas still produce incorrect figures; determinism only guarantees execution of the code. The paper uses Qwen3-235B-Instruct and Gemini3-family backbones, with separate Qwen3, Gemini-3-Flash, and Gemini-3-Pro ImgCoder entries in the main table.

3. Visual Necessity and Hybrid Evaluation: check individual facts and whole-image usability

Atomic questions are derived from source instructions, not from the surface content of generated images. Gemini-3-Flash extracts numerical values, geometric relationships, or domain properties and turns each into a single-fact question. GPT-5-nano then answers without an image in 4 independent trials. A question answered correctly in all 4 trials is removed to reduce textual leakage and commonsense shortcuts. This does not prove that every remaining question is impossible to answer blindly; it only establishes that it failed this repeated blind-solving test.

Within each image type, selection favors samples with more valid atomic questions, followed by expert checks of logic and facts. Gemini-3-Flash answers these questions using the generated image, and the image passes inverse validation only when every associated question is answered correctly. The following standard notation reconstructs the prose definition of Eq. (2); the cached equation has broken formatting, so this is not a verbatim transcription:

\[ R_{\mathrm{inv}}=\frac{1}{|\mathcal D|}\sum_{I\in\mathcal D}\mathbf{1}\!\left[\sum_{q\in Q_I}V(I,q)=|Q_I|\right]. \]

Here \(\mathcal D\) is the evaluation image set, \(Q_I\) contains an image's atomic questions, \(V(I,q)\in\{0,1\}\) indicates a correct visual answer, and \(\mathbf{1}\) is an indicator. Tables report this proportion as a percentage; it is not pooled question-level accuracy. Requiring all answers to be correct exposes missing critical relationships, but also introduces sensitivity to the number of questions per image and the visual question-answering model's ability.

A parallel evaluation branch asks Gemini-3-Flash to assign 0, 1, or 2 points on five dimensions: Correctness & Fidelity, Layout & Precision, Readability & Occlusion, Scientific Plausibility, and Expressiveness & Richness. These distinguish a logically precise but sparse diagram from a rich image with incorrect quantitative relationships. PSNR, SSIM, CLIP, and FID serve as auxiliary references rather than arbiters of scientific correctness.

4. Downstream Utility Testing: prevent image-text training from becoming text-only solving

Training examples must retain reliable scientific supervision while making images necessary for solving. The authors mask some informational cues in the original problem text so that the model must recover them from the image instead of solving a nominally multimodal task from text alone. This serves a different purpose from blind filtering: the former modifies training inputs, while the latter filters evaluation questions.

The downstream comparison trains the same Qwen3-VL-8B-Instruct model on data from different generators, adding Nanobanana-Pro (Filt) and Qwen-Image (Filt) subsets with incorrect images removed. This tests whether synthesis paradigm, teacher quality, and filtering translate into reasoning gains. Not every training group is filtered, so it would be inaccurate to claim that all groups receive equally stringent image verification.

A Worked Example

Figure 2 uses \(y=x\ln x\) to illustrate precision differences. A pixel model may draw a smooth curve that resembles a function plot while misplacing intercepts and extrema. ImgCoder identifies the function and axes during understanding, specifies the plotting region and labels during planning, and then computes and plots function values through code. Correctness depends on the function and coordinate calculations, not texture detail.

Atomic questions derived from source facts then test whether the diagram's relationships can be recovered. Failure to read one necessary relationship is enough to fail image-level inverse validation, even when labels are clear and the picture looks polished. Conversely, code rendering can lose expressiveness by omitting contextual detail from a spring system; the other example in Figure 2 illustrates why programmatic synthesis does not dominate pixel generation on every dimension.

Loss & Training

ImgCoder is an inference workflow for understanding, planning, and code generation; the main text introduces no dedicated training loss for it. Downstream experiments use rollout-based reinforcement learning implemented in VeRL for 200 global steps, with multiple rollouts per prompt and Compass-Verifier-8B for unified answer verification and reward computation.

Evaluation uses MathVisionmini and Geometry3Ktest and averages performance over multiple samples; Figure 5 labels accuracy as Avg@4. The available cache contains the main paper and references but not the cited supplementary material, so learning rate, batch size, exact rollout count, RL objective, and hardware settings cannot be supplied as established facts.

Key Experimental Results

Main Results

The following entries come from Table 2. \(R_{\mathrm{inv}}\) is a percentage, higher is better; the remaining columns are Gemini-3-Flash judge scores on a 0-2 scale, also higher is better. The abbreviations denote Correctness & Fidelity, Layout & Precision, Readability & Occlusion, Scientific Plausibility, and Expressiveness & Richness.

Method	\(R_{\mathrm{inv}}\) (%)	C&F	L&P	R&O	SP	E&R
Qwen-Image	38.86	0.24	0.70	1.48	0.30	0.76
GPT-Image-1.5	63.52	0.98	1.70	1.97	1.17	1.62
Nanobanana-Pro	73.41	1.59	1.87	1.98	1.72	1.93
Qwen3-ImgCoder	56.38	1.21	1.30	1.62	1.39	1.29
Gemini-3-Flash-ImgCoder	76.93	1.80	1.88	1.88	1.92	1.91
Gemini-3-Pro-ImgCoder	77.87	1.82	1.93	1.91	1.93	1.90

Gemini-3-Pro-ImgCoder exceeds Nanobanana-Pro by 4.46 percentage points, but scores slightly lower on readability and expressiveness. Qwen3-ImgCoder exceeds Qwen-Image by 17.52 percentage points; these are not the same model with only its renderer changed, so this comparison cannot isolate the net causal contribution of code execution.

The downstream comparison below comes from Table 6. GEO3K and MV report answering performance on a percentage scale. AVG is preserved as printed and must not be silently replaced with the simple mean of the two columns.

RL Data	GEO3K	MV	AVG (Reported)
Base	61.9	39.0	54.5
Qwen-Image	68.2	45.9	57.1
Qwen-Imgcoder	67.9	46.9	57.4
Qwen-Image (Filt)	68.6	47.0	57.8
Nanobanana-Pro	70.1	46.1	58.1
Gemini-ImgCoder	69.1	46.9	58.0
Nanobanana-Pro (Filt)	68.7	47.7	58.2

The paper reports a 3.7-point gain in AVG for the best group over Base. However, the arithmetic mean of the Base scores 61.9 and 39.0 is 50.45, not 54.5, and the main text does not explain the discrepancy. These values are retained with their aggregation left unresolved. The directly checkable MV change is from 39.0 to 47.7, an 8.7-percentage-point increase.

Ablation Study

The following entries come from Table 5. \(R_{\mathrm{inv}}\) retains the table's precision, Tokens denotes reported token usage, and Runtime is in seconds. The caption says "Planned ablation of ImgCoder components," while the prose discusses observed results, so the values are cited as reported rather than as independently reproduced experiments.

Config	\(R_{\mathrm{inv}}\) (%)	Tokens	Runtime (s)
Gemini3-F-ImgCoder	76.9	7412	37.5
w/o Plan	74.8	3663	18.7
w/o Code	75.5	3205	16.9
w/o Plan & Code (Direct)	74.3	3046	16.1

The full pipeline exceeds direct generation by 2.6 percentage points while adding 4366 tokens and 21.4 seconds. Planning and code both have positive associations with performance, but inference budgets also change. This is not a fixed-compute estimate of component contributions, and the available cache does not fully specify the code-free implementation or timing conditions.

Key Findings

• The evaluator affects the winner: replacing it with GPT-5-mini in Table 4 changes Gemini-3-Pro-ImgCoder's inverse validation rate from 77.87% to 75.16%, while Nanobanana-Pro rises from 73.41% to 77.31%. High overall ranking correlation does not mean every rank is unchanged.
• Human evaluation covers 500 samples with 10 STEM annotators. Elo starts at 1500, uses a scale parameter of 400 and update coefficient of 32, and is averaged over 100 shuffled comparison orders. Figure 4 reports Pearson correlations of 0.957 and 0.911 between human Elo and inverse validation rate and mean judge score, respectively.
• Figure 5c scales Nanobanana-Pro synthetic training data from 50 to 1400 samples, improving MV Avg@4 from 43.9% to 46.1%, or 2.2 percentage points. This supports growth within the tested range, not a scaling law established for unlimited extrapolation.

Highlights & Insights

• Requiring recovery of all necessary information in an image matches the purpose of scientific training data better than image similarity. The five-dimensional judge adds diagnostic detail so a single pass rate does not hide the distinction between label occlusion and domain-knowledge errors.
• ImgCoder's value lies in specifying content, coordinates, label anchors, and scientific constraints, not simply asking a model to emit code. This is suitable for charts, circuits, and geometry, but the plan itself still needs validation.
• Connecting data quality with training gains takes generation evaluation beyond a leaderboard. Filtering improves MV but does not consistently improve GEO3K, highlighting the need to check data augmentation utility separately for each task.

Limitations & Future Work

• The authors discuss a precision-expressiveness trade-off: code-generated figures tend to be schematic, while pixel models support richer scenes. Hybrid methods are promising, but this paper does not validate a complete hybrid generation system.
• Inverse validation depends on a visual question-answering model and varying question counts per image. More questions increase opportunities for one recognition failure to reject an entire image. Cross-family judges and human preferences reduce single-evaluator risk without proving scientific correctness for every sample.
• The unmatched compute budgets in Table 5, the anomalous AVG reporting in Table 6, and the unavailable supplement limit strict reproduction. These results do not establish that code generation must outperform alternatives at fixed cost.
• The experiments cover only 200 RL steps and 50-1400 samples for data scaling. Larger training runs, variance reporting, and parser- or solver-based verification of topology and quantitative constraints would provide stronger evidence than adding another model judge alone.

Related Work & Insights

• vs AutomaTikZ / VisCoder: these methods generate scientific graphics or executable visualizations through code. This paper places the programmatic route within a unified scientific-information evaluation and downstream training protocol; ImgCoder does not originate programmatic drawing.
• vs ScImage / SridBench: the former tests spatial, numerical, and attribute understanding in scientific contexts, while the latter targets research illustrations. SciGenBench emphasizes strict scientific relationships and preservation of information necessary for reasoning.
• vs TIFA / SeePhys: TIFA represents question-based image-faithfulness evaluation, while SeePhys studies the role of vision in physics reasoning. This paper combines source-fact questions, blind filtering, an all-questions-correct image metric, and training utility to assess reliable usability rather than appearance alone.

Rating

• Novelty: 4/5. Unifying scientific image evaluation and utility testing is valuable; the generation framework itself is primarily a structured prompting baseline.
• Experimental Thoroughness: 3/5. Model coverage, human assessment, and cross-family evaluation are broad, but compute controls and numerical reporting leave gaps.
• Writing Quality: 3/5. The research chain is clear, while table captions, AVG reporting, and supplementary settings limit verifiability.
• Value: 4/5. Useful for designing and validating scientific multimodal data, rather than establishing unconditional superiority of code-based generation.