FineEdit: Fine-Grained Image Edit with Bounding Box Guidance

Conference: ECCV 2026
Paper: ECCV Official
Code: https://laulampaul.github.io/FineEdit
Area: Object Detection
Keywords: Diffusion Model, Image Editing, Bounding Box Guidance, Multi-level Injection, Decoupled Reinforcement Learning

TL;DR

FineEdit employs early channel-wise concatenation and deep side-adapter feature injection to embed bounding-box spatial priors into a diffusion transformer, complemented by a decoupled foreground-background reinforcement learning post-training strategy to deliver precise localized editing without background leakage.

Background & Motivation

Diffusion-based image editing models have demonstrated impressive generative capabilities across diverse real-world domains. However, conventional editing pipelines rely almost exclusively on natural language instructions to designate both the target location and the desired modification. In complex, multi-object scenarios, textual descriptions inherently suffer from spatial ambiguity, struggling to uniquely anchor the intended subject. Consequently, because these diffusion architectures operate on a global regeneration paradigm, unconstrained denoising frequently alters non-targeted regions, leading to severe background drifting, color shifts, and accidental structural distortion.

Integrating visual spatial priors into diffusion architectures remains an active yet challenging pursuit. Generic structural conditioning methods, such as ControlNet, typically apply holistic spatial constraints that lack the targeted isolation necessary for isolated object editing. Inpainting-based alternatives, such as RePaint or BrushNet, are fundamentally tailored for pixel-level content reconstruction and missing-region completion rather than complex, open-vocabulary semantic editing; they often fail to follow nuanced compositional prompts and suffer from boundary artifacts or box-edge leakage. Meanwhile, recent editing systems that attempt to take bounding-box conditions often yield boundary bleed-through or box residual outlines, and their post-training reinforcement learning stages rely on coarse global VLM ratings that entangle target compliance with background preservation, confusing the optimization gradients.

To overcome these structural limitations, FineEdit introduces an intuitive yet rigorous framework that exploits sparse bounding-box priors to explicitly delineate editing boundaries while preserving global integrity. The core idea is to inject multi-level bounding-box priors through early input channel concatenation and deep side-adapter features across intermediate diffusion transformer layers, followed by a decoupled post-training reinforcement learning strategy that independently optimizes localized instruction alignment and background pixel/perceptual retention.

Method

Overall Architecture

FineEdit builds upon the Qwen-Image backbone, utilizing Qwen2.5-VL to encode multimodal conditional representations from the text prompt and the source image. An MM-DiT diffusion transformer processes both noisy target image latents \(\hat{q}_{\text{dst}}\) and clean source image latents \(q_{\text{src}}\) extracted via a frozen VAE. The framework is trained in two coordinated stages: a pre-training stage that establishes multi-level spatial guidance via early channel concatenation and deep adapter injection, followed by a post-training stage that optimizes editing quality using decoupled reinforcement learning rewards.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Source Image x_src + Bounding Box b + Text Prompt c"] --> B["Early Channel Injection: Masked Image Latent Concatenated with Target Latent"]
    B --> C["Deep Feature Injection: 6-layer Side-Adapter Injected into Intermediate Blocks"]
    C --> D["MM-DiT Backbone Denoising & Velocity Prediction"]
    D --> E["Decoupled RL Post-Training: Separate ROI Semantic & Background Preservation Rewards"]
    E --> F["Output: Edited Target Image x_dst with Preserved Background"]

Key Designs

1. Early Channel Injection: Grounding Spatial Boundaries in Initial Latent Manifolds To prevent edits from spilling over into non-target areas, the model explicitly incorporates bounding-box spatial constraints into the input latent space. Given a bounding box \(b = [x_1, y_1, x_2, y_2]\), a binary mask \(M \in \{0, 1\}^{H \times W}\) is constructed where pixels outside the box are set to 1 and those inside are set to 0. An element-wise masked image \(x_{\text{msk}} = M \odot x_{\text{src}}\) is encoded by the frozen VAE into a masked latent \(q_{\text{msk}}\). In the denoising transformer input, \(q_{\text{msk}}\) is concatenated along the channel dimension with the noisy target latent \(\hat{q}_{\text{dst}}\) as \([\hat{q}_{\text{dst}} \parallel q_{\text{msk}}]\), while the clean source latent \(q_{\text{src}}\) is padded with a zero tensor to maintain identical channel dimensions. This channel-wise fusion explicitly informs the model of the exact background pixel context alongside the exact boundary of the region of interest (ROI) from timestep zero.

2. Deep Feature Injection: Multi-Scale Spatial Representation via Lightweight Side-Adapter Relying solely on early channel concatenation often leads to spatial signal dilution as representations propagate through dozens of self-attention blocks. FineEdit introduces a 6-layer lightweight Transformer side-adapter (\(\mathcal{A}\)), which introduces only 10% additional parameters relative to the base diffusion transformer. The side-adapter encodes the masked latent \(q_{\text{msk}}\) into deep feature representations and injects them uniformly into intermediate layers of the MM-DiT backbone. For the \(i\)-th block, the updated feature \(h_i\) is expressed as:

\[h_i = \mathcal{F}_i([ [ q_{\text{src}} \parallel 0 ], [ \hat{q}_{\text{dst}} \parallel q_{\text{msk}} ] ] \mid t, c) + \mathcal{A}(q_{\text{msk}})\]

This deep injection acts as a cross-layer skip connection that preserves fine-grained localization cues throughout the entire feature hierarchy, stabilizing the generation of intricate details while accelerating model convergence.

3. Decoupled RL Post-Training: Disentangled Foreground Compliance and Background Preservation Rewards Conventional RL fine-tuning for image editing utilizes holistic, full-image VLM preference scores, which fail to distinguish between subtle foreground modifications and unintended background perturbations. FineEdit decomposes the overall reward function into two independent objectives:

\[R(x_{\text{dst}}) = R_{\text{roi}} \times R_{\text{bg}}\]

For the target region, \(R_{\text{roi}}\) leverages Qwen3-VL to evaluate whether the local transformation precisely aligns with the text instruction within the bounding box. For the background region, \(R_{\text{bg}}\) combines objective reconstruction fidelity with perceptual naturalness:

\[R_{\text{bg}} = \alpha \cdot \Psi_{\text{PSNR}}(\bar{M} \odot x_{\text{src}}, \bar{M} \odot x_{\text{dst}}) + (1 - \alpha) \cdot \Phi_{\text{VLM}}(x_{\text{dst}}, \bar{M}, c_2)\]

where \(\bar{M} = 1 - M\) is the inverted background mask, \(\Psi_{\text{PSNR}}\) computes normalized peak signal-to-noise ratio over non-edited pixels, and \(\Phi_{\text{VLM}}\) rates background consistency. By optimizing LoRA parameters (\(r=32, \alpha=64\)) against this factored objective, the model avoids reward hacking and achieves both high semantic fidelity and background invariance.

4. Unified Visual Instruction Paradigm & U-GAF Dataset Engine: Versatile Editing and High-Fidelity Data Curation The bounding-box interface acts as a flexible controller across diverse editing granularities. Shrinking or expanding the bounding box enables seamless transitions between local additions/replacements, multi-box composite edits, and full-canvas global style transfer (\(B_{\text{global}} \approx I\)), including fine-grained localized style transfer where only the target object adopts a new artistic style while its background remains untouched. To power this architecture, the authors introduced the U-GAF (Unified Generation, Annotation, and Filtering) pipeline, which uses Co-DETR for detection and enforces a strict four-stage filtering cascade (difference mask IoU verification, RGB entropy filtering, dual-scale Qwen3-VL global/local semantic checks, and EditScore artifact filtering). This pipeline yielded FineEdit-1.2M, a curated dataset of 1.2 million high-quality pairs that eliminates synthetic misalignment.

Loss & Training

Pre-training is conducted across 64 NVIDIA H100 GPUs using the AdamW optimizer with a learning rate of \(5 \times 10^{-5}\) at \(1024 \times 1024\) resolution. A per-device batch size of 1 with 8 gradient accumulation steps establishes an effective global batch size of 512 over 15,000 steps. Post-training freezes the backbone and fine-tunes LoRA adapters (\(r=32, \alpha=64\)) at a learning rate of \(3 \times 10^{-4}\), employing an RL scheme organized into 24 groups per epoch with 16 samples per group for advantage estimation.

Key Experimental Results

Main Results

On the FineEdit-1k Benchmark, FineEdit outperforms prior state-of-the-art methods in both background preservation and localized editing accuracy:

Model	Background PSNR ↑	Background SSIM ↑	Background LPIPS ↓	Out-of-Box Retention OBR ↑	BBox CLIP ↑	Prompt Compliance PC ↑	Visual Naturalness VN ↑	Physical & Detail Integrity PDI ↑
Flux-fill-dev	34.2	0.90	0.08	4.99	0.26	3.56	4.03	3.98
BrushNet	29.6	0.83	0.13	4.99	0.25	3.28	3.38	3.34
BrushEdit	30.9	0.84	0.13	4.95	0.25	3.27	3.45	3.41
PrefPaint	27.1	0.77	0.21	4.83	0.23	2.74	3.01	2.96
Asuka-flux	34.4	0.88	0.16	5.00	0.22	2.20	2.92	2.85
FluPA	32.7	0.89	0.09	4.96	0.26	3.68	4.05	4.01
GLM-Image	27.4	0.85	0.14	4.48	0.25	3.66	3.98	3.95
Z-Image-turbo-Control	28.7	0.86	0.10	4.99	0.24	3.18	3.51	3.48
LongCat-Image-Edit	27.3	0.83	0.12	4.38	0.26	4.59	4.52	4.56
Qwen-Image-Edit	31.4	0.88	0.11	4.23	0.25	3.94	4.21	4.19
Qwen-Image-Edit-2509	27.1	0.80	0.13	4.15	0.26	4.56	4.66	4.65
FineEdit (Pre-trained)	34.4	0.91	0.08	4.80	0.27	4.67	4.71	4.69
FineEdit-r1 (RL Tuned)	34.7	0.91	0.09	4.89	0.27	4.80	4.83	4.81

On GEdit-Bench-EN, FineEdit achieves 8.71 in G_SC and an overall score of 7.97, outperforming Qwen-Image-Edit-2509 (7.54) and GPT-Image-1 [High] (7.53). On ImgEdit-Bench, FineEdit attains an overall score of 4.50, surpassing Qwen-Image-Edit-2509 (4.35) and GPT-4o-Image (4.20).

Ablation Study

The impact of multi-level fusion strategies and reward formulations on FineEdit-1k is evaluated below:

Ablation Config	Background PSNR ↑	Background SSIM ↑	Out-of-Box Retention OBR ↑	BBox CLIP ↑	Prompt Compliance PC ↑	Visual Naturalness VN ↑	Physical & Detail Integrity PDI ↑	Note
Baseline (Qwen-Image-Edit-2509)	27.1	0.80	4.15	0.26	4.56	4.66	4.65	Vanilla baseline without multi-level injection
Early Fusion Only	15.9	0.48	2.43	0.21	2.45	1.85	1.85	Severe convergence failure from input distribution shock
Deep Fusion Only	33.1	0.88	4.61	0.26	4.52	4.51	4.61	Preserves backbone manifold but lacks sharp boundary grounding
Early + Deep Fusion	34.4	0.91	4.80	0.27	4.67	4.71	4.69	Complete pre-trained architecture
Pre-RL Checkpoint	34.4	0.91	4.80	0.27	4.67	4.71	4.69	Baseline prior to reinforcement learning
Global Reward RL	32.7	0.90	4.76	0.27	4.74	4.76	4.73	Global score sacrifices background preservation (-1.7 dB PSNR)
Decoupled Reward RL	34.7	0.91	4.89	0.27	4.80	4.83	4.81	Decoupled objectives improve both ROI and background metrics

Key Findings

• Relying exclusively on early fusion results in acute optimization divergence (PSNR drops to 15.9 and PC to 2.45) due to severe disturbance of the pre-trained latent space manifold. Deep fusion via the side-adapter acts as a crucial regularizer, stabilizing optimization while boosting convergence speed.
• Conventional global VLM rewards induce an unwanted trade-off: enhancing instruction adherence slightly while degrading background fidelity (PSNR degrades from 34.4 to 32.7). In contrast, decoupled rewards enhance both foreground editing fidelity (PC rises to 4.80) and background retention (PSNR reaches 34.7 dB).
• Bounding-box scale analysis indicates robust stability across box-area ratios ranging from tiny details (3%) to expansive scenes (100%), preventing target overlooking in small regions and distortion in large areas.

Highlights & Insights

• Dual-Stream Early Channel and Deep Adapter Synergy: Solves the fundamental trade-off between strict boundary localization and latent manifold stability, adding only 10% parameter overhead while preventing boundary leakage.
• Decoupled Reward Function for Spatial RL: Separating local ROI instruction compliance from non-edited pixel-perceptual retention addresses reward misalignment, providing a clean optimization paradigm for region-constrained generative models.
• Unified Visual Instruction Formulation: Elegantly handles single-object edits, multi-box composite tasks, and localized style transfer without requiring separate task-specific heads or complex inference-time test-time optimizations.

Limitations & Future Work

• The framework currently relies on rectangular bounding boxes as its spatial conditioning primitive. For non-convex, curved, or disconnected objects (e.g., thin cables, fine hair strands, lattice structures), bounding boxes inevitably encompass neighboring background pixels.
• Extending visual conditioning to more fine-grained interactive prompts, including points, scribbles, and polygon contours, represents a promising future avenue to achieve sub-pixel geometric precision.

Related Work & Insights

• vs Qwen-Image-Edit & LongCat-Image-Edit: Existing instruction-driven editors accept bounding boxes but suffer from boundary bleed-through and background degradation. FineEdit achieves a 3-7 dB gain in background PSNR and substantially higher human-rated prompt adherence through multi-level feature injection and decoupled RL.
• vs BrushNet & BrushEdit: Inpainting models are optimized for reconstruction and struggle with complex semantic transformations or stylization. FineEdit retains full semantic generative expressiveness while establishing clean spatial boundaries.

Rating

• Novelty: ⭐⭐⭐⭐☆ An elegant combination of early and deep spatial injection with decoupled reinforcement learning rewards for localized editing.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation on the dedicated FineEdit-1k benchmark as well as open benchmarks (GEdit-Bench and ImgEdit-Bench), complete with multi-dimensional low-level and high-level evaluations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, cohesive structure, and insightful ablation analyses.
• Value: ⭐⭐⭐⭐⭐ Addresses the critical problem of background corruption in diffusion editing, providing a high-quality 1.2M dataset and reusable architectural principles.