Skip to content

SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing

Conference: ICCV 2025 arXiv: 2505.02370
Area: Diffusion Models ยท Image Editing Keywords: instruction-based editing, supervision rectification, contrastive learning, diffusion prior, VLM, triplet loss

TL;DR

SuperEdit addresses the noisy supervision problem in instruction-based image editing by leveraging diffusion generation priors to guide a VLM in rectifying editing instructions, and by constructing contrastive supervision signals (positive/negative instructions + triplet loss), surpassing SmartEdit by 9.19% with less data and a smaller model.

Background & Motivation

Training data for instruction-based image editing is typically generated by automated pipelines (LLM rewrites captions โ†’ diffusion model generates edited images), but diffusion models cannot precisely follow text instructions, leading to:

  • Mismatches between edited images and editing instructions
  • Unintended modifications to regions that should remain unchanged
  • Noisy supervision signals

Limitations of existing approaches:

Scaling data (InstructPix2Pix): the noisy supervision problem remains unresolved

Introducing large VLMs (SmartEdit, MGIE): prohibitive computational cost (14.1B parameters)

Pre-training on recognition tasks (InstructDiffusion): indirect mitigation that does not address the root cause

SuperEdit's key insight: the problem lies in the supervision signal itself, not the model architecture. Rectifying instructions is more direct and effective than scaling up models.

Method

1. Diffusion Generation Prior

Core finding: editing models generate fixed attributes independent of text at different inference stages: - Early stage: global layout - Middle stage: local object attributes - Late stage: image details - Style changes: span all stages

This prior provides a unified foundation for instruction rectification.

2. Rectifying Supervision

GPT-4o is used to regenerate accurate editing instructions from originalโ†’edited image pairs according to the four generation attributes (global/local/detail/style).

Procedure: 1. Feed the original and edited images to GPT-4o 2. Guide the VLM to describe differences according to the four change categories defined by the diffusion prior 3. Aggregate descriptions into a rectified instruction that precisely matches the image pair 4. Ensure instruction length does not exceed the 77-token limit of the CLIP text encoder

3. Facilitating Supervision (Contrastive Supervision)

Even after instruction rectification, the editing model still struggles to distinguish semantically similar instructions (e.g., "add a cat on the left" vs. "add two cats on the right").

Constructing contrastive instructions: GPT-4o substitutes a single attribute (quantity/position/object) in the rectified instruction to generate a negative instruction \(c^T_{neg}\).

Triplet Loss:

\[\mathcal{L}_{\text{triplet}} = \max\{d(\epsilon_t, \epsilon_{pos}) - d(\epsilon_t, \epsilon_{neg}) + m, 0\}\]

where: $\(\epsilon_{pos} = \epsilon_\theta(\text{concat}(x_t, c^I), t, c^T_{pos})\)$ $\(\epsilon_{neg} = \epsilon_\theta(\text{concat}(x_t, c^I), t, c^T_{neg})\)$

Total Loss:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{train}} + \lambda \cdot \mathcal{L}_{\text{triplet}}\]

where \(\mathcal{L}_{\text{train}} = d(\epsilon_t, \epsilon_{pos})\) is the standard diffusion loss.

Key Experimental Results

Comparison on the Real-Edit Benchmark

Method Extra Module Pre-training Data Size Model Size Overallโ†‘
SmartEdit โœ“ โœ“ 1.2M 14.1B 3.59
MGIE โœ“ โœ“ 1.0M 8.1B 2.86
InstructPix2Pix โœ— โœ— 300K 1.1B 3.31
SuperEdit โœ— โœ— 40K 1.1B 3.92

SuperEdit surpasses SmartEdit by 9.19% using 30ร— less data and a 13ร— smaller model.

GPT-4o Automatic Evaluation

Method Following Accโ†‘ Preserving Accโ†‘ Quality Accโ†‘ Overall Accโ†‘
SmartEdit 64% 66% 45% 58.3%
SuperEdit 75% 72% 55% 67.3%

SuperEdit achieves comprehensive improvements across instruction following, content preservation, and image quality.

Ablation Study

Configuration Followingโ†‘ Preservingโ†‘ Qualityโ†‘ Overallโ†‘
Original instructions (baseline) 52% 53% 50% 51.7%
+ Rectified instructions 70% 68% 52% 63.3%
+ Rectified instructions + contrastive loss 75% 72% 55% 67.3%

Instruction rectification contributes approximately 11.6% improvement; the contrastive loss provides an additional 4%.

Highlights & Insights

  1. Data-centric rather than model-centric: improving supervision signals rather than scaling up models yields greater gains at minimal cost
  2. Generality of the diffusion prior: editing models and T2I models share the same stage-wise generation attributes
  3. Elegant application of contrastive learning: modifying only a single attribute in the instruction ensures small embedding distance but large semantic difference between positive and negative samples
  4. Fully open-sourced (data + model) with strong reproducibility

Limitations & Future Work

  • Relies on GPT-4o for instruction rectification and contrastive instruction generation, incurring API costs
  • Inherent limitations of the InstructPix2Pix architecture (e.g., resolution constraints of SD 1.5)
  • The contrastive loss requires an additional UNet forward pass per step
  • Integration with stronger base models (SDXL, Flux) remains unexplored
  • Instruction-based editing: InstructPix2Pix, MagicBrush, SmartEdit
  • Editing data construction: Prompt-to-Prompt, EditBench
  • Diffusion model alignment: DPO for diffusion, ReFL

Rating

Dimension Score (1โ€“5)
Novelty 4
Technical Depth 4
Experimental Thoroughness 5
Writing Quality 4
Overall 4.2