GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing¶

Conference: ECCV2026
Paper: ECCV Paper
PDF: Full Paper
Project: GRADE
Code: https://github.com/VisionXLab/GRADE
Dataset: VisionXLab/GRADE
Area: Vision-Language Reasoning / Image Editing
Keywords: disciplinary knowledge, implicit reasoning, image editing, automated evaluation, structural constraints

TL;DR¶

GRADE turns image editing into a joint test of disciplinary knowledge and diagram constraints, evaluating 20 models on 520 samples with three scoring dimensions; the best strict accuracy is only 46.2%, while explicit editing instructions substantially reduce but do not eliminate failures.

Background & Motivation¶

Unified multimodal models aim to combine understanding, reasoning, and image generation, but producing a plausible image does not establish that these capabilities work together. Conventional editing tasks explicitly request object replacement, color changes, or background adjustments, allowing a direct mapping from language to appearance. Even when instructions contain implicit conditions, many existing evaluations rely on everyday commonsense rather than professional knowledge that must be applied procedurally. RISEBench and KRISBench already introduce reasoning into editing evaluation, but their knowledge coverage differs from tasks involving academic diagrams. For example, drawing a reaction product from a molecular diagram requires determining the chemical transformation before implementing it through the correct bonds and atoms.

Such tasks constrain both the answer and its representation: a correct reaction mechanism still does not yield a valid edit if the model corrupts an unchanged molecular backbone. Rotation axes in geometry, pitches in music notation, and missing positions on historical timelines likewise require interpreting the disciplinary meaning of visual elements. Image-text similarity and visual appeal can overlook misplaced symbols, incorrect labels, and impossible spatial relationships. Understanding benchmarks such as MMMU and HLE test knowledge without usually requiring the answer to be drawn back into the source image; generation benchmarks such as MMMG and GenExam do not impose the same pressure to preserve input structure. The gap addressed here is whether a model can correctly modify a professional diagram under constraints after understanding it, not how to design a new editing network.

GRADE selects human-curated editing tasks across ten disciplines and gives each task a reference result and individually checkable knowledge scoring points. Evaluation therefore asks whether the required changes occurred, unrelated content remained intact, and the result can be interpreted correctly, rather than merely whether the output resembles a target. The main focus is implicit instructions: the model receives the disciplinary goal without being handed every graphical operation. Explicit instructions appear only as an auxiliary ablation condition for assessing how failures relate to knowledge-based derivation. Core Idea: use disciplinary knowledge to define the editing target, then jointly assess reasoning correctness, task-dependent consistency, and logical readability to test whether understanding becomes reliable visual modification.

Method¶

Overall Architecture¶

GRADE stands for Grounded Reasoning Assessment for Discipline-informed Editing and provides a dataset and evaluation protocol, not a model to train. Each sample contains an input image, a textual editing instruction, and a ground-truth reference image (GT); the evaluated model receives the first two and returns an edited image. The reference image supports evaluation rather than serving as an answer directly supplied to the editor in the standard task. Data preparation establishes disciplinary tasks, expert-reviewed knowledge scoring points, and task-specific consistency categories. At evaluation time, Gemini-3-Flash scores the model output, after which dimension scores, strict accuracy, and a relaxed score are reported separately. The four key designs are disciplinary task construction, knowledge scoring points, task-dependent visual constraints, and dual-level aggregation and diagnosis. These intermediate representations are sample annotations and evaluation rules, not new latent variables, neural branches, or training-supervision modules. Accordingly, the contribution does not need a network architecture diagram, and GRADE introduces neither a training loss nor a claimed improvement from fine-tuning a model.

Key Designs¶

1. Disciplinary task construction: make the target depend on expertise rather than surface matching

The dataset contains 520 samples spanning mathematics, physics, chemistry, biology, history, geography, sports, music, computer science, and economics. For most samples, six annotators with relevant academic backgrounds collect concept-grounded images from open textbooks, websites, and other reference materials. They manually create input-reference pairs and corresponding instructions, followed by cross-validation by two additional experts. The remaining samples come from MMMU: an automated pipeline performs coarse filtering, two experts select samples and design instructions, and two further experts cross-validate them. Manual editing here is intended to establish a reference result tied to a checkable disciplinary requirement, not to manufacture more visual styles. Unlike relying solely on language-model-generated questions, this procedure subjects the correspondence among image, instruction, and correct edit to expert review. The main paper does not report the number of samples from each source, so the dataset cannot be described as entirely independent of existing benchmarks.

Each discipline also has finer second-level categories, such as plane geometry, solid geometry, functions, graphs, and statistics within mathematics. This supports overall comparison while allowing errors to be examined against particular knowledge structures. Instructions generally retain a derivation step, for instance asking for a reaction product without listing the chemical bonds to remove or add. Similarly, asking for the correct shortest path in a graph does not specify every edge that should be recolored. Consequently, perception, knowledge selection, reasoning steps, and final drawing jointly determine the result, rather than merely instruction-following at the level of graphical operations. Ten-discipline coverage does not imply equal sample counts or equal difficulty, so aggregate scores should not be treated as an unbiased ranking of professional competence.

2. Knowledge scoring points: turn correctness into verifiable binary conditions

Discipline Reasoning asks whether the edited result correctly reflects the required knowledge, rather than whether the model produces a plausible explanation. GPT-5 generates weighted binary questions for each sample, with weights summing to 1. Two experts with relevant disciplinary knowledge independently inspect every scoring point, followed by cross-validation by a third expert. These questions are prepared before formal evaluation, avoiding ad hoc changes to success criteria across models. During evaluation, Gemini-3-Flash uses the scoring points and reference image to judge whether the edited result satisfies each condition. GPT-5 therefore prepares the rubric, whereas Gemini-3-Flash applies it to outputs; these are distinct stages.

In the shortest-path task in Figure 1, scoring points check whether three target edges are red, whether they form a continuous path from A to H, and whether redundant red edges are absent. Each of the five conditions receives weight 0.2, distinguishing a few correctly drawn edges from a complete solution. Writing the answer to question \(k\) as \(b_k\in\{0,1\}\) and its weight as \(w_k\), the verbal definition in Section 3.2 can be expressed as:

\[ R=\sum_{k=1}^{K}w_k b_k,\qquad \sum_{k=1}^{K}w_k=1. \]

Here \(R\) ranges from 0 to 1; the equation formalizes the paper's scoring definition and is not a new optimization objective. The reference image helps identify the target state, while binary questions make required disciplinary properties explicit, reducing the risk that a generic judging instruction omits them. However, finer questions do not automatically eliminate judge bias: rubric coverage, weight choices, and the judge's ability to recognize small symbols still affect results.

3. Task-dependent visual constraints: distinguish what to preserve from whether the answer is readable

Visual Consistency cannot simply reward resemblance to the input, since some correct edits must change the global structure. GRADE divides tasks into three categories, established through annotator consensus, and applies corresponding evaluation prompts. Localized consistency covers edits to specified regions or elements, such as completing a timeline or adjusting curves in an economics diagram, while leaving other elements unchanged. Style consistency permits structural changes but preserves representation style, such as retaining bond-line notation after modifying a molecule instead of switching to a ball-and-stick rendering. Consistency independence covers tasks that need not retain the input appearance, such as producing engineering orthographic views from a rendered mechanical part. The third category does not waive representational standards; it means those standards come from the target format rather than a universal similarity requirement. Each sample receives a consistency score of 0, 1, or 2; the main paper does not provide the full prompts or detailed grade boundaries for every category.

Logical Readability also uses scores of 0, 1, or 2, but concerns whether disciplinary information can be clearly interpreted. An image with sharp outlines and sufficient resolution may still be unusable because curves cannot be distinguished, legends are missing, or labels contradict one another. The judge considers logical clarity, annotations and legends, spelling, and consistency of representational conventions. Reasoning correctness thus concerns whether the right knowledge is expressed, while logical readability concerns whether a reader can reliably interpret that expression. Separating them helps distinguish an incorrect answer from a badly rendered answer rather than collapsing every failure into poor generation quality. Visual Consistency additionally constrains editing scope, preventing a model from bypassing the task by drawing an unrelated but correct diagram. Conversely, barely changing the original image may yield high consistency and readability, so these dimensions must be used jointly with Discipline Reasoning.

4. Dual-level aggregation and diagnosis: measure both complete success and partial progress

Strict accuracy counts a sample as successful only when all three dimensions achieve their maximum scores. With \(C,L\in\{0,1,2\}\) denoting consistency and readability, respectively, the rule over \(N\) samples is:

\[ \mathrm{Accuracy}=\frac{100}{N}\sum_{i=1}^{N}\mathbf{1}[R_i=1\land C_i=2\land L_i=2]. \]

This metric tests complete delivery, so an output missing a necessary label or adding one extra edge may fail. It is not another name for the reasoning score, and an average reasoning score of 77.5 cannot be interpreted as a 77.5% rate of fully successful edits. To avoid hiding partial improvements behind the strict threshold, the authors also normalize each dimension to 0-100 and combine them with weights 0.6, 0.3, and 0.1. The relaxed score is therefore 60% normalized Discipline Reasoning, 30% Visual Consistency, and 10% Logical Readability. Strict accuracy addresses whether an output is ready to use, while the relaxed score tracks partial satisfaction but allows compensation between dimensions.

Judge reliability is checked on 68 samples drawn across disciplines, using outputs from Nano Banana Pro, GPT-Image-1.5, and Qwen-Edit-2511. Five experts follow the same protocol, their averaged scores form the human reference, and each dimension is assessed using mean absolute error (MAE) and standard deviation (STD). A separate diagnostic keeps input and reference images fixed while converting implicit instructions into explicit descriptions of the required knowledge-driven edits. This intervention reduces the burden of deriving operations, but still requires image recognition and correct drawing; it is not a causal experiment that isolates every variable. Error analysis combines Nano Banana Pro outputs with the chain-of-thought (CoT) reported in the paper to distinguish image recognition, knowledge, reasoning-process, and generation-process errors. These are qualitative diagnostic categories; the main paper gives no benchmark-wide proportions, so the examples cannot establish an overall error distribution.

Key Experimental Results¶

Main Results¶

Table 1 on page 9 evaluates 20 model configurations, including 10 closed-source and 10 open-source configurations; representative results are selected below. Dimension scores are normalized to 0-100, and accuracy is the percentage achieving maximum scores in all three dimensions under implicit instructions on the full 520-sample benchmark.

Model	Discipline Reasoning	Visual Consistency	Logical Readability	Strict Accuracy (%)
Nano Banana Pro	77.5	89.5	95.8	46.2
Nano Banana 2	72.6	86.4	95.9	39.6
Seedream 5.0	64.1	87.5	90.6	24.7
GPT-Image-1.5	54.5	82.3	90.7	16.0
Qwen-Edit-2511	18.6	45.2	52.1	2.7
Step-1X (think+reflect)	19.2	57.2	66.9	2.3
FLUX.2 dev	23.0	56.4	69.0	2.1
DreamOmni	17.4	83.2	89.1	1.0

Nano Banana Pro exceeds GPT-Image-1.5 by 30.2 percentage points, yet still fails to satisfy every requirement on more than half the samples. Qwen-Edit-2511 is the best open-source configuration by strict accuracy, not by the reasoning score in Table 1; FLUX.2 dev scores 23.0 on the latter metric. DreamOmni's consistency of 83.2 and readability of 89.1 contrast with its accuracy of 1.0, which the authors associate with minimal or absent changes to the input. Table 3 on page 11 reports a relaxed score of 82.9 for Nano Banana Pro, illustrating the large gap between partial correctness and complete success.

Ablation Study¶

Table 5 on page 13 compares implicit and explicit instructions on a 68-sample subset, with input and reference images unchanged; score units match the main table. The subset baseline differs from the full benchmark, so its 35.3 should not be compared directly with the main table's 39.6 to infer an instruction effect.

Model	Instruction	Discipline Reasoning	Visual Consistency	Logical Readability	Strict Accuracy (%)
Nano Banana 2	Implicit	67.9	83.8	95.6	35.3
Nano Banana 2	Explicit	89.7	93.0	95.5	65.7
Qwen-Edit-2511	Implicit	18.9	40.0	47.8	1.5
Qwen-Edit-2511	Explicit	44.7	60.0	81.6	8.8

Strict accuracy increases by 30.4 percentage points for Nano Banana 2 and 7.3 for Qwen-Edit-2511; the latter gains more in relative terms but retains low absolute success. Reasoning improves by 21.8 and 25.8 points, respectively, supporting implicit knowledge derivation as an important bottleneck without establishing that generation execution is solved. Nano Banana 2 readability decreases from 95.6 to 95.5, so the ablation cannot be summarized as an improvement in every dimension for every model.

Key Findings¶

The judge analysis in Table 4 on page 12 uses the human-alignment setup described above, with scores normalized to 0-1; lower MAE/STD is better throughout.

Judge Model	Reasoning MAE / STD	Consistency MAE / STD	Readability MAE / STD
Qwen3-VL-235B	0.1798 / 0.3504	0.1231 / 0.3523	0.0838 / 0.2944
GPT-5	0.1519 / 0.3227	0.1677 / 0.4386	0.1221 / 0.3638
Gemini-3-Flash	0.1194 / 0.2834	0.0954 / 0.3047	0.0838 / 0.2944

Gemini-3-Flash has the lowest errors for reasoning and consistency, but ties Qwen3-VL-235B for readability rather than strictly outperforming every judge in every column. The timeline example in Section 4.4 also shows closed-source models selecting the wrong insertion location while open-source models may select the right location but supply the wrong event, so failures do not lie on a single capability axis. Source inconsistency: the prose on page 10 gives FLUX.2 dev consistency as 17.6%, whereas Table 1 gives 56.4; this note preserves the table value and flags the discrepancy without silently correcting the source.

Highlights & Insights¶

Knowledge scoring points diagnose partial correctness, while the strict threshold demands complete delivery. Together they reduce the risk of overestimating usability from averages alone.
Task-dependent consistency is more appropriate than universally requiring input preservation. Local editing and engineering-view conversion should not share one appearance-preservation rule.
Explicit-instruction intervention provides a practical diagnostic direction. Readers can adapt it to test target derivation and editing execution separately, but the paper does not validate the effectiveness of a new two-stage system.

Limitations & Future Work¶

Coverage and statistical stability are limited: 520 samples span ten disciplines, and the main paper reports neither confidence intervals for major rankings nor repeated-generation variance, making small differences difficult to interpret.
Judge errors remain: human alignment on 68 samples supports local reliability, but reasoning MAE is 0.1194 and does not establish reliable strict-success decisions across the full benchmark.
The all-maximum threshold can amplify borderline scoring errors, while the relaxed score can reward conservative non-editing; practical assessment should inspect dimension scores and representative failures together.
The available material is the main paper; the complete secondary taxonomy, judging prompts, and detailed collection procedures are deferred to supplementary material and are not reconstructed here.
Reader-proposed extensions include symbolic checkers for geometry or graph constraints and cross-judge, cross-run stability analysis; the paper does not evaluate these approaches.

Compared with ImgEdit: ImgEdit emphasizes conventional edits with explicit operations, whereas GRADE makes deriving operations from a disciplinary goal the central difficulty rather than merely expanding operation types.
Compared with RISEBench / KRISBench: these benchmarks examine reasoning-oriented editing; GRADE moves toward professional knowledge and structured-diagram constraints, contributing a task axis and protocol rather than replacing all commonsense editing evaluations.
Compared with MMMU / HLE / GenExam: the first two emphasize understanding and GenExam emphasizes disciplinary generation; GRADE adds targeted modification under existing input structure, requiring correct knowledge to become correct visual operations.
Research implication: auditing perceptual parsing, knowledge solving, and constrained drawing separately may locate problems more effectively than merely extending CoT; this is a suggestion derived from the four error categories, not a new experimental result.

Rating¶

Novelty: 4/5. The combination of disciplinary knowledge and editing constraints gives the benchmark a clear purpose, with novelty concentrated in evaluation design.
Experimental Thoroughness: 4/5. It covers 20 configurations with human alignment and instruction ablations, but subset size and uncertainty reporting remain limited.
Writing Quality: 3/5. The core protocol is clear, but prose-table numerical conflicts and imprecise summaries of dimension-wise leadership weaken reporting.
Value: 4/5. It offers interpretable diagnostics for professional diagram editing, but is not a comprehensive reliability certification for real disciplinary workflows.