RePlan: Reasoning-Guided Region Planning for Complex Instruction-Based Image Editing

Conference: ECCV2026
Paper: ECCV Paper
Official ID: 3410
Citation Count: 12 (provided with the task, 2026-09-17; not independently verified online)
Area: Vision-Language Reasoning / Image Generation
Keywords: instruction-based image editing, region planning, chain-of-thought, attention region injection, GRPO

TL;DR

RePlan uses a vision-language model to determine which regions to edit and how, then executes the edits jointly through attention region injection without retraining the editor, improving Flux.2-Klein-9B Overall from 3.45 to 3.92 on IV-Edit, primarily through better regional consistency.

Background & Motivation

Instruction-based image editing must identify the correct target among similar objects, not merely generate an attractive result. Replacing a cup that has been used and left on a desk with a plant requires reasoning about its contents, apparent state, and the instruction, rather than replacing every cup. The paper calls the difficulty arising from complex scenes, complex instructions, and their interaction Instruction-Visual Complexity (IV-Complexity).

Methods such as InstructPix2Pix learn a direct mapping from instructions to edited images; Qwen-Image and Bagel further introduce vision-language capabilities, but correct understanding does not guarantee spatially correct generation. Encoding an entire request as global semantic conditioning can still expose similar objects to the same edit. Sequential region-based inpainting strengthens localization but requires repeated execution, propagating errors and image degradation between rounds.

RePlan therefore focuses on the interface between understanding and editing: besides specifying the desired appearance, it explicitly assigns each hint to a region while preserving global visual interaction. Core Idea: represent reasoning as inspectable bounding-box and editing-hint pairs, then route each hint to its intended region through attention visibility rules instead of asking diffusion to infer the target again from global text.

Method

Overall Architecture

The input is an image and a natural-language editing instruction; the output is the edited image. The Region-Aligned Planner generates chain-of-thought reasoning, a global hint, and local bounding-box/hint pairs. Attention Region Injection separately encodes the hints and restricts their interaction with image tokens according to regions. Rule-Group Attention executes these constraints efficiently. All regions are generated jointly in one editing process rather than in sequential inpainting rounds.

The default planner is Qwen2.5-VL 7B, and the editing backbone used during training is Flux.1-Kontext-dev. GRPO trains the planner, whereas attention injection itself requires no diffusion-editor retraining; the complete RePlan system is therefore not entirely training-free. The paper also applies the framework to Qwen-Image-Edit and Flux.2-Klein-9B.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and editing instruction"] --> Planner["Region-Aligned Planner"]
    Planner --> Injection["Attention Region Injection"]
    Injection --> Acceleration["Rule-Group Attention"]
    Acceleration --> Output["Jointly denoised edited image"]

Here, a single pass means one joint editing process, not a single diffusion denoising step. The framework retains the pretrained editor's generation procedure but avoids repeating the entire procedure for every region.

Key Designs

1. Region-Aligned Planner: resolve references before producing an executable plan

The planner first combines image understanding with the implicit conditions in the instruction, translating abstract requests into concrete visual changes. It outputs structured text separated by <think>, <global>, and <region>: reasoning, a global hint, and a JSON list whose entries contain bbox_2d and hint. These fields support interpretation and allow downstream components to extract coordinates and editing actions directly.

The global hint covers the whole image and may request a background change or preservation of the remaining content. Local hints describe changes for individual targets. A local hint can also explicitly preserve an object. This is not a conventional diffusion negative prompt that suppresses particular concepts; it associates a preservation instruction with a specific region to protect nearby objects that might be confused with the target.

An explicit plan makes failure attribution easier: a wrong box points to a planning problem, while an incorrect output despite a correct box and hint points to execution. Users can also modify boxes or hints directly. GRPO improves both plan parseability and actual editing outcomes, with the rewards described under Loss & Training.

2. Attention Region Injection: isolate local language conditions while preserving global visual interaction

The editor uses MMDiT to place text, source-image features, and noisy target latents into a unified attention sequence. RePlan encodes the global and regional hints separately before concatenating their tokens, rather than mixing all hints into one text representation. The source image is VAE-encoded into a patch grid; bounding boxes define local image-token groups, and patches outside all local groups form the background group.

Visibility follows five rules: full interaction within a group; no visibility between different text-hint groups; global interaction among image and latent tokens; access from each local image group only to its own hint and the global hint; and access from the background only to the global hint. The purpose is not to split the image into independent crops, but to remove incorrect language-conditioning connections while retaining visual information flow.

This distinction explains how local control can coexist with a coherent appearance. A region does not directly receive another region's recoloring hint, but visual tokens can still coordinate illumination, boundaries, and style. Bounding boxes are not hard guarantees of unchanged pixels outside the box: the paper explicitly allows effects to extend beyond them when necessary. The available main text does not fully specify conflict resolution for overlapping boxes or contradictory hints.

3. Rule-Group Attention: replace dense masks with on-the-fly visibility checks

Materializing a dense token-to-token mask is straightforward but can impede efficient attention kernels and increase memory traffic. Here, visibility depends on a small number of rule groups, typically fewer than 10, rather than on a unique rule for every token pair. Storing the full pairwise matrix is therefore unnecessary.

Each token stores two bitsets: the groups it belongs to and the groups it may attend to. During attention, a bitwise AND between the query token's allowed groups and the key token's membership groups determines visibility through a nonempty intersection. FlexAttention fuses this test into the kernel, preserving the semantics of region injection without materializing a dense mask.

The paper reduces additional storage from \(O(n^2)\) to \(O(n\lceil k/W\rceil)\) machine words, where \(n\) is the token count, \(k\) the rule-group count, and \(W\) the word size in bits. This concerns mask-related storage, not a claim that all attention computation becomes linear. The benefit is especially useful with multiple regions because joint editing mainly lengthens the hints instead of regenerating the entire image for each target.

A Worked Example

The desk example in Figure 3 contains a red coffee cup and a glass holding water. Asked to replace the cup that has been used and left with a small plant, the planner selects the red cup because it appears empty while the glass still contains water. This is an inference about the visible scene, not knowledge of the objects' actual usage histories.

The plan assigns [224, 372, 263, 431] to the red cup and requests replacement with a plant. It assigns [175, 329, 220, 388] to the glass and requests preservation, while the global hint preserves the remaining content. After separate encoding, red-cup patches receive the replacement hint, glass patches receive the preservation hint, and background patches receive the global preservation instruction.

During joint denoising, hints remain isolated but visual tokens communicate globally so that the plant can fit the desk's appearance. If the glass changes incorrectly, one can first inspect whether the planner produced the correct protection region, then inspect hint visibility, instead of relying solely on repeated rewrites of the entire instruction.

Loss & Training

The paper uses approximately 1k complex editing examples consisting of source images and instructions, without paired target edited images; the main text also mentions supervised alignment on these examples before RL. GRPO compares multiple candidate plans and their outcomes for the same instruction and updates the Qwen2.5-VL 7B planner. These examples are not used to train a new diffusion backbone.

The first stage makes plans executable. Tag-format rewards check the marker structure; region-format rewards check whether the JSON list and dictionaries parse correctly. The so-called reasoning-quality reward increases with the length of the text inside <think>, up to a cap. Length is only a proxy for reasoning behavior, not direct verification of correctness, regardless of the reward's name.

The second stage decodes plans into images and uses Qwen2.5-VL 72B to evaluate Target, Effect, and Consistency: correct target selection, fulfillment of the requested visual change, and preservation of irrelevant regions and global style. To discourage obtaining high consistency by making no edits, the paper multiplies consistency reward by effect reward:

\[ R'_C = R_C \cdot R_E. \]

The second-stage objective combines target, effect, and reweighted consistency rewards while retaining the first-stage format reward with a small weight. Some equations are damaged by text extraction, and the main text does not give numerical values for this weight or reward normalization; unverifiable hyperparameters are not supplied here. The training evaluator, Qwen2.5-VL 72B, must also be distinguished from the test evaluator, Gemini-2.5-Pro.

Key Experimental Results

Main Results

IV-Edit contains around 800 image-instruction pairs covering 7 referring types and 16 task types. Instructions average 21 words, and 182 examples involve multiple target regions. Gemini-2.5-Pro assigns 1-to-5 ratings for Quality, Target, Effect, and Consistency; Overall is their simple average. Target and Consistency are rating dimensions, not standard detection recall and precision.

The following values are selected from Table 1, with higher being better throughout. Weighted is intended to modulate Consistency by Effect, but the product formula in the main text does not clearly specify scale handling and cannot be recomputed directly from the four column means. Only the reported table values are retained here, without inventing a formula.

Model	Quality	Target	Effect	Consistency	Overall	Weighted
Gemini-Flash-Image	3.89	4.11	3.93	2.89	3.71	3.44
Flux.1-Kontext-dev	3.93	3.34	2.73	2.88	3.22	2.49
Flux.1-Kontext-dev + RePlan	4.16	3.47	2.59	3.64	3.46	2.55
Qwen-Image-Edit	3.47	3.72	3.24	1.79	3.05	2.62
Qwen-Image-Edit + RePlan	3.86	3.77	3.16	3.24	3.51	2.91
Flux.2-Klein-9B	4.05	3.81	3.31	2.61	3.45	2.95
Flux.2-Klein-9B + RePlan	4.28	4.04	3.41	3.96	3.92	3.33

Ablation Study

Table 2 uses Flux.1-Kontext-dev as the editor, with the full method achieving Overall/Weighted of 3.46/2.55. These are absolute values from the table; zero-shot planner results should not be mistaken for results with different editing backbones.

Config	Overall	Weighted	Note
Gemini2.5-Pro zero-shot planner	2.95	1.93	Authors observe bounding-box errors
Qwen2.5-VL 7B zero-shot planner	2.60	1.63	Weaker hint decomposition and format compliance
Without reasoning	3.31	2.49	Directly outputs regional plans
Single-stage RL	3.42	2.51	Skips first-stage format learning
RePlan full model	3.46	2.55	Reasoning and two-stage RL

Key Findings

• Consistency improves most reliably: the three backbones rise from 2.88, 1.79, and 2.61 to 3.64, 3.24, and 3.96. However, Effect drops from 2.73 and 3.24 to 2.59 and 3.16 for the first two backbones, so gains are not universal across editing capabilities.
• Flux.2-Klein-9B + RePlan reaches Overall 3.92 versus Gemini-Flash-Image's 3.71, but its Weighted 3.33 remains below 3.44. Claims of surpassing proprietary models must name the metric.
• Table 3 reports five-region end-to-end latency of 221.0 seconds for multi-turn Kontext, 113.8 seconds for naive masked RePlan, and 54.1 seconds for Rule-Group Attention RePlan. For one region, the accelerated method and baseline both take 44.2 seconds. These values apply to the paper's setup, not arbitrary hardware.
• In a blinded comparison of 100 examples by 12 experts, RePlan wins 35%, Kontext wins 15%, and 50% are ties. The 70% win rate applies only after excluding ties, not to the complete sample.

Highlights & Insights

• An explicit plan is a debuggable intermediate representation. Boxes and hints separate target-understanding failures from execution failures and allow localized user corrections.
• Regional control need not isolate image regions completely. Isolating language conditions while retaining visual communication supports both local constraints and global coherence.
• Algorithmic constraints and kernel implementation reinforce each other. Fusing group rules into attention makes training-free masking practical without excessive execution overhead.

Limitations & Future Work

• The main text has no dedicated limitations section. The following points are reading-based assessments of the method and results, not claims explicitly attributed to the authors.
• The planner can make localization, implicit-semantic, and formatting errors; a length reward cannot establish reasoning faithfulness. Region verification or output feedback could help, but the paper does not demonstrate a complete automatic correction loop.
• Pixels outside a box are not guaranteed to remain unchanged, and Effect falls for some backbones. Dense overlapping objects, conflicting local hints, and strict pixel-preservation tasks need dedicated evaluation.
• Weighted scale and aggregation details remain unclear, and automatic evaluation depends on an external VLM. The 100-example human comparison covers only the Kontext combination, not all backbones or metrics.
• The available cache includes the main paper and references, but not the appendix mentioned in the text. Full data sources, reward hyperparameters, and detailed training settings cannot be verified from it. Approximately 1k examples refer only to planner adaptation, not the foundation models' pretraining costs.

Related Work & Insights

• vs Qwen-Image / Bagel: the contribution is not another jointly trained understanding-generation model, but a regional execution interface for existing VLM reasoning. The interface is pluggable, while final visual quality and editing capacity still depend on the backbone.
• vs multi-turn inpainting: sequential methods edit regions one after another; RePlan handles them jointly in one editing process. Its latency advantage becomes clearer with more regions, but it is not a few-step diffusion algorithm.
• vs KrisBench / RISEBench: IV-Edit emphasizes interactions between complex scenes and complex references, including text, tables, and posters. Reasoning difficulty should not be considered separately from target-localization difficulty.
• Resources: ECCV official paper page and project page listed in the paper. The project URL comes from the paper and was not checked online; no code repository or arXiv ID is inferred.

Rating

• Novelty: 4/5. A clear interface between regional plans and constrained attention, complemented by practical rule-group acceleration.
• Experimental Thoroughness: 4/5. Multiple backbones, planner and training ablations, latency measurements, and human comparison, with remaining metric and reproducibility limitations.
• Writing Quality: 4/5. Clear motivation and execution rules, but some evaluation scales and implementation boundaries need clarification.
• Value: 4/5. Useful for complex instructions and multi-region editing, without establishing a universal replacement for general image editors.