UniREditBench: A Unified Reasoning-based Image Editing Benchmark

Conference: ECCV2026
Paper: ECCV Paper
Project: UniREditBench
Area: Image Generation / Reasoning-based Image Editing
Keywords: dual-reference evaluation, visual reasoning, multi-object interaction, game rules, chain-of-thought

TL;DR

UniREditBench extends image editing evaluation to real-world interactions and game-rule reasoning, constrains the evaluator with textual and target-image references, and trains UniREdit-Bagel with synthetic chain-of-thought data to achieve an in-domain overall score of 78.15, while remaining behind the strongest closed-source models on out-of-distribution RISEBench.

Background & Motivation

Image editing models can change colors, replace materials, and remove objects, but a visible change does not necessarily satisfy the causal implications of an instruction. An instruction to draw a paddle backward through water requires understanding interactions among the paddle, water, and canoe, rather than merely relocating the paddle. A Sokoban board additionally requires obeying human-defined rules, such as boxes not passing through walls or being pulled. Both settings demand that understanding of the scene and its rules be realized in the visible edited state. Traditional benchmarks emphasize local attributes and appearance; although RISEBench and KRISBench introduce reasoning evaluation, this paper argues that multi-object interactions and game scenarios remain insufficiently covered.

A second problem lies in evaluation: both the editing model and the vision-language model (VLM) judging it can make mistakes. With only a target description, the evaluator might notice a continuous red line in a maze yet overlook an unnecessary detour into a dead end. In the example in Figure 2, the output receives a score of 5 with textual references alone and 3 after adding the target image. This does not make an image reference automatically authoritative; it shows that natural-language descriptions can leave complex spatial details insufficiently checked. Increasing task difficulty therefore needs to be accompanied by validation of the scoring protocol, or leaderboard improvements may partly reflect evaluation error.

The paper accordingly connects evaluation with training: it first defines a broader task space and dual-reference protocol, then supplies supervision through corresponding synthesis procedures. Real-world images are constructed through textual planning and generative models, whereas programs produce legal game states and answers instead of relying on the same image-generation process for both domains. The synthetic data also contains chain-of-thought (CoT) explanations of editing consequences, allowing Bagel to learn both target images and the textual process connecting initial and target states. Core Idea: organize image editing around whether physical consequences or rules are correctly rendered, evaluate outputs against both text and target images, and use scalable reasoning traces for training.

Method

Overall Architecture

Rather than introducing a new editing architecture, the paper provides UniREditBench, UniREdit-Data-100K, and a training validation based on Bagel. The benchmark contains 2,700 samples, each with an input image, an editing instruction, a textual description of the target effect, and a target-image reference. At test time, an editing model produces an output from the input image and instruction; reference answers are used for judging, not supplied as default extra inputs to the model being tested. Evaluation separates instruction following, visual consistency, and visual quality before applying fixed aggregation weights. The separate training dataset contains 100,421 samples and augments image-editing pairs with stepwise reasoning text. UniREdit-Bagel learns to generate a reasoning trace followed by an edited image, testing whether the data improves both reasoning and visual execution.

The design has four parts: task coverage, dual-reference evaluation, scenario-specific data synthesis, and joint reasoning-text and image supervision. Task coverage defines which abilities to assess, dual-reference evaluation defines how to judge results, and the remaining parts provide training examples and test their usefulness. The evaluator, synthesis models, and editing model should not be interpreted as one inference network: they belong to evaluation, offline data preparation, and model training, respectively. This distinction also matters for interpreting results: in-domain improvement establishes effectiveness on related tasks, whereas transfer requires separate out-of-distribution evidence.

Key Designs

1. Task coverage: extend object changes to interaction consequences and explicit rules

The real-world portion requires not only object recognition but also rendering the structural, motion-related, or medium-mediated consequences of an operation. Section 3.2 divides single-object transformations into viewpoint transformation, pose adjustment, temporal evolution, and material modification. Multi-object interactions cover structural integrity change, motion state change, mechanical reaction, medium interaction, and spatial arrangement. For example, inserting a needle into a balloon can change the balloon's integrity, not merely the needle's position; operating a mechanism should likewise change the relevant component states. This design prevents a model from succeeding merely by depicting the nouns in an instruction. It must determine what changes, how those changes proceed, and which background elements and attributes remain untouched.

The game-world portion covers long-horizon planning, logical puzzle solving, strategic reasoning, and spatial intelligence. Maze and Sokoban examine multi-step paths and state transitions, while Sudoku, Tic-tac-toe, and Word Search examine formal constraints and symbolic relations. Pacman, Jewel2, and Space Invader provide different action or strategy rules, while 3D Reconstruction tests spatial structure understanding. The paper reports 8 primary dimensions and 18 subcategories, each containing 150 human-inspected benchmark examples. The narrative grouping in Section 3.2 does not fully match the real-world column names in Table 2, which uses aggregate labels including attribute modification, structure transform, physical interaction, and property response. This note therefore preserves the stated coverage and counts without inventing an exact mapping between the two taxonomies.

2. Dual-reference evaluation: separate target fulfillment, preservation, and visual quality

GPT-4.1 assigns instruction-following scores, IF, after inspecting the original image, instruction, generated image, target image, and textual target description together. The textual reference explains the intended effect, while the image reference supports direct comparison of spatial layouts and visible outcomes. Visual consistency, VC, uses the original image, instruction, and output to check whether unrelated regions or attributes changed unintentionally. Visual quality, VQ, examines the generated image for artifacts, distortions, and physical or logical implausibility. These criteria are not interchangeable: regenerating the whole scene may look attractive but destroy identity or background, while preserving the original image may leave the instruction unfulfilled. Dual references primarily enter the IF judgment; VC and VQ do not automatically receive the same additional reference inputs.

All three raw scores range from 1 to 5, and Section 3.3 explicitly gives the aggregation rule:

\[ S_{\mathrm{Overall}}=0.5S_{\mathrm{IF}}+0.3S_{\mathrm{VC}}+0.2S_{\mathrm{VQ}}. \]

The largest weight goes to instruction following, prioritizing task completion over appearance while still penalizing unrelated changes and low-quality generation. The main text connects weight selection to agreement with human ratings, but the detailed weight ablation is in an appendix absent from the supplied cache. The main results table uses a different numerical scale, and the readable main text does not specify its conversion from the 1-to-5 scores. The results below are therefore reported as table scores, without interpreting 78.15 as a success rate or guessing a linear rescaling formula. Human evaluation uses 6 experts, 200 random samples, and 4 models, measuring disagreement through mean absolute error, MAE; lower is better.

3. Scenario-specific data synthesis: semantic generation for natural images, program constraints for games

Real-world synthesis follows a text-then-image process beginning with hand-written triples: an original-image description, an editing instruction, and an expected-effect description. Gemini 2.5 Pro expands these seeds, and GPT-4o synthesizes the original and edited target images from the resulting descriptions. This establishes the intended before-and-after semantic relationship before image generation rather than inferring an edit from arbitrary similar images. A VLM subsequently checks visual fidelity, instruction alignment, and hallucination risks, and generates CoT text for accepted instances. The house-of-cards case in Figure 4 illustrates the supervision: disturbing the bottom supports displaces lower cards and starts a structural collapse while the background and lighting remain unchanged. These target images come from a synthesis process and should not be mistaken for records of physical experiments.

Game data instead begins with problem definitions and rules, after which category-specific Python programs generate initial states, edited states, instructions, and textual references. Programs also emit traceable reasoning steps, providing explanations tied to state transitions rather than only answer images. A VLM converts the programmatic traces into natural-language CoT, followed by quality filtering. Compared with natural-image synthesis, this route assigns legal moves and constraint satisfaction to programs, making it suitable for boards, mazes, and rule-based games. Benchmark examples additionally undergo VLM filtering and human inspection; each training category contains more than 4,000 instances to broaden supervision. These controls do not make all synthetic content unbiased, especially when generation and screening models may share blind spots.

4. Joint reasoning-text and image supervision: learn to render consequences, not just describe them

Each training example consists of an original image, an instruction, a stepwise CoT trace, and the target edited image. The model first predicts reasoning text conditioned on the original image and instruction, then generates an image conditioned on the image, instruction, and reasoning. Text supervision uses negative log-likelihood to match the synthetic explanation, while image supervision uses a latent flow-matching objective. The objectives constrain the explanation and visual output separately, rather than training a model that can explain an edit but cannot execute it. This also differs from adding a request to think step by step at inference time: the reasoning traces and target images both drive parameter updates. Reasoning text is a supervised intermediate representation during training, not a reference answer inserted into the model during evaluation.

The cached text of Section 4.2 has substantial corruption in the text loss, flow-matching loss, and combined-loss equations. Readable prose establishes the objective types, conditioning inputs, and joint-supervision relationship, but not a faithful reconstruction of the complete expressions. This note therefore does not invent the authors' exact flow-matching path or expanded loss, and retains only clearly readable training settings. In particular, the damaged latent-path expression should not be treated as evidence of a novel editing algorithm. The discussion mentions an analysis of CoT, but that appendix is unavailable, and the visible results cannot independently quantify the benefit of reasoning supervision relative to the additional data.

Loss & Training

Training updates every Bagel component except the VAE for 5,000 iterations with Adam. The cosine learning-rate schedule uses 500 warm-up steps, a peak of \(2\times10^{-5}\), and a minimum of \(10^{-6}\). Text and image losses have weights \(\lambda_{\mathrm{text}}=2\) and \(\lambda_{\mathrm{img}}=1\), respectively. UniREdit-Bagel uses the official Bagel inference configuration, while other baselines retain their original inference configurations. This compares models under their established usage settings rather than matching test-time compute, resolution, or latency budgets.

Key Experimental Results

Main Results

The following selection comes from Table 2, page 11, on UniREditBench with GPT-4.1 as evaluator; higher is better. Real-world and game-world scores retain the original scenario averages, and Overall retains the reported aggregate without recomputing its aggregation.

Model	Real World	Game World	Overall
GPT-4o	81.01	62.07	73.39
Nano Banana	75.22	60.39	68.26
Qwen-Image-Edit	70.95	41.92	56.52
Bagel	56.60	39.42	48.01
Bagel-Think	56.80	45.10	50.96
UniREdit-Bagel	75.74	80.48	78.15

UniREdit-Bagel exceeds Bagel-Think by 27.19 overall points and GPT-4o by 4.76, but remains below GPT-4o on real-world scenarios. For game-world scenarios, the table values 80.48 and 62.07 differ by 18.41 points, whereas Section 5.3 states +17.08; this note flags the inconsistency rather than correcting the source results.

The following selection comes from Table 3, page 13, on out-of-distribution RISEBench, also evaluated by GPT-4.1; percentage signs follow the source. The cache does not fully specify the decision thresholds underlying these percentages, so their magnitudes should not be directly compared with or subtracted from the preceding table's scores.

Model	Temporal	Causal	Spatial	Logical	Overall
GPT-4o	34.1%	32.2%	37.0%	10.6%	28.9%
Nano Banana	25.9%	47.8%	37.0%	18.8%	32.8%
Qwen-Image-Edit	4.7%	10.0%	17.0%	2.4%	8.9%
Bagel	3.5%	4.4%	9.0%	5.9%	5.8%
Bagel-Think	4.7%	15.5%	14.0%	1.2%	9.2%
UniREdit-Bagel	22.4%	18.9%	21.0%	10.6%	18.3%

Out-of-distribution overall performance increases from Bagel-Think's 9.2% to 18.3%, a gain of 9.1 percentage points. It remains below GPT-4o at 28.9% and Nano Banana at 32.8%, so the in-domain lead does not establish superiority across all reasoning-based editing tasks.

Ablation Study

The following is the evaluator-reliability analysis from Table 4, page 14, not a training-component ablation; it reports MAE between human and GPT-4.1 ratings, with lower values better. The study uses 6 experts and 200 random samples; IF, VC, and VQ denote instruction following, visual consistency, and visual quality.

Evaluated Model	IF MAE	VC MAE	VQ MAE	Average MAE
Qwen-Image-Edit	0.73	0.62	0.47	0.61
Nano Banana	0.85	0.42	0.66	0.64
GPT-4o	0.42	0.68	0.22	0.44
UniREdit-Bagel	0.66	0.54	0.51	0.57

Figure 7 compares dual references, no image reference, and no textual reference; the authors report the lowest MAE for dual references at every human-score level. The cached chart does not provide reliably readable point values, so its visual trend is not converted into invented numerical ablation entries. The main text refers weight selection, game-data transfer, CoT, and scaling from 20K to 100K examples to the appendix; configuration-specific differences are not fabricated here.

Key Findings

• Gains are concentrated in game tasks without requiring a universal lead in real-world editing; GPT-4o retains the higher real-world average.
• Out-of-distribution results support some transfer, but closed-source models retain a clear advantage, so in-domain task matching cannot substitute for generalization evaluation.
• Evidence that dual references reduce evaluation error comes from human comparisons, not just model rankings; MAE below 1 does not establish that every individual judgment is correct.

Highlights & Insights

• Requiring rules to be satisfied in the resulting image is closer to editing needs than checking whether a model states the right answer; game states make these constraints amenable to programmatic synthesis.
• Separating target fulfillment from preservation exposes outputs that look attractive as new images but are not faithful edits of the original.
• The synthesis routes exploit different strengths: natural scenes broaden semantic coverage, while programs align states with rules instead of forcing all data through one generator.

Limitations & Future Work

• Reader assessment: training data and the in-domain benchmark share synthesis ideas and task categories, so distribution matching can influence scores; the cached main text is insufficient to verify a complete deduplication and leakage audit.
• Reader assessment: a single target image may favor one valid solution, especially for open-ended real-world consequences; multiple references or executable rule checks could avoid penalizing different but correct edits.
• Reader assessment: synthesis, filtering, and judging depend on strong VLMs, and a small human-MAE study cannot exclude correlated biases, although it does not establish that favoritism occurred either.
• Source-evidence boundary: damaged loss equations, an unavailable appendix, conflicting game-world gains, and unspecified score rescaling limit conclusions about exact reproducibility details.

Related Work & Insights

• Compared with RISEBench: it evaluates temporal, spatial, and logical reasoning in editing; UniREditBench broadens interaction and game coverage and uses RISEBench to examine cross-benchmark transfer.
• Compared with KRISBench: it organizes factual, conceptual, and procedural knowledge; this paper emphasizes scenario and rule coverage plus textual and visual references, while its stated KRISBench transfer results are absent from the cached appendix material.
• Compared with Bagel-Think: the baseline already supports thinking before editing; this work adds domain-relevant reasoning traces and target-image supervision, so its gains cannot be reduced to merely enabling a thinking mode.
• Research direction: programmatic checkers could verify game-output legality while VLMs judge perceptual quality and open-ended semantics; this is an extension suggested by the paper, not an experiment already demonstrated in it.

Rating

• Novelty: 4/5. The combination of task coverage and dual-reference evaluation has clear value, while the model largely reuses Bagel.
• Experimental Thoroughness: 4/5. In-domain, out-of-distribution, and human-rating analyses are included, but the unavailable appendix limits verifiable training ablations.
• Writing Quality: 3/5. The problem is clear, but inconsistent category descriptions and gain figures complicate precise interpretation; cached equation extraction is also damaged.
• Value: 4/5. Useful for studying rule execution, reasoning-data synthesis, and evaluator reliability in image editing, with results interpreted alongside the transfer gap.