SIMON: SImultaneous Multi-Object Navigation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yijichar/SIMON-SImultaneous-Multi-Object-Navigation
Area: Image Generation
Keywords: Image Editing, Controllable Diffusion Models, Multi-Instance Editing, Training-Free, Energy-Guided Diffusion

TL;DR¶

SIMON tackles the critical challenges of attention dispersion, ghosting artifacts, and multiplicative inference latency in multi-object image editing through a training-free framework featuring content-aware attention, multi-object energy guidance, and latent initialization, achieving superior visual fidelity and constant inference efficiency in a single reverse diffusion trajectory.

Background & Motivation¶

Fine-grained instance-level image editing using pre-trained diffusion models has emerged as a cornerstone of modern generative vision. While existing training-free drag and editing approaches (such as DragonDiffusion, DiffEditor, and GeoDiffuser) achieve remarkable spatial manipulation on single objects or isolated facial benchmarks, they degrade severely when confronted with real-world indoor and outdoor scenes containing multiple interacting objects and cluttered backgrounds. When users attempt to reposition multiple instances simultaneously or perform complex spatial reorganizations like object shuffling, existing pipelines resort to sequential single-object editing. This sequential execution inevitably incurs error accumulation: the inpainting and relocation of the earliest objects directly distort the background or reference regions required by subsequent objects, leading to severe spatial conflicts, boundary blurring, and corrupted scene layouts.

The fundamental technical dilemma lies in attention dispersion and latent memory contamination inherent to multi-instance diffusion dynamics. In standard cross- and self-attention operations, queries corresponding to target instances inadvertently distribute attention across all visually similar entities in the scene, diluting editing precision. Furthermore, DDIM inversion caches latents ($z_T$) that tenaciously preserve the high-frequency structural and textural traces of original objects inside their vacated regions. Conventional feature-matching guidance struggles to erase these deeply rooted priors, leaving noticeable ghosting artifacts. Finally, handling multiple instances independently without a unified energy formulation causes mutual competition across inpainting, object relocation, and background preservation subtasks, while scaling runtime linearly with object count.

The core insight of SIMON is to abandon naive sequential processing in favor of a synchronized, spatially disentangled paradigm that concurrently addresses three coupled subtasks: inpainting vacated holes, faithfully relocating target objects, and preserving untouched background. Core idea: by combining task-mask-based content-aware attention scaling, a multi-object composite energy guidance field, and Gaussian latent initialization with early background anchoring, SIMON achieves synchronized multi-object manipulation and seamless hole inpainting within a single reverse diffusion path, ensuring high identity fidelity and near-constant inference runtime.

Method¶

Overall Architecture¶

The SIMON pipeline builds upon pre-trained latent diffusion models and DDIM inversion. Given an input image $x_0$ alongside $N$ target objects defined by bounding boxes and target displacement vectors $c_n = [cx_n, cy_n]$, off-the-shelf segmentation models generate binary masks $m_n$ for initial object footprints and $\hat{m}_n$ for translated target destinations. These spatial configurations define three mutually complementary task masks: the untouched background $m^{\text{bg}} = \complement \left( \bigcup_{n=1}^N m_n \cup \hat{m}_n \right)$, the vacated inpainting region for each object $m_n^{\text{ipt}} = m_n \setminus \bigcup_{j=1}^N \hat{m}_j$, and a surrounding contextual reference region $m_n^{\text{ref}}$ providing background inpainting cues.

During reverse diffusion sampling, SIMON first initiates Latent Initialization by wiping inverted latents in vacated inpainting zones with standard Gaussian noise, while clamping the background to original inversion trajectories during initial timesteps. In each denoising step, Content-aware Attention intervenes in self-attention layers by modulating query-key interactions via multiplicative scaling and additive biasing across task-designated regions. Concurrently, Multi-Object Energy Guidance constructs a composite objective spanning all three subtasks, computing gradients with respect to the generated latents to steer denoising trajectories toward desired layouts in parallel.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x0 with Multi-Object Annotations<br/>Derive Initial Masks mn and Target Masks m^n"] --> B["DDIM Inversion & Feature Caching<br/>Store zT and Attention Key/Values Kt_ori, Vt_ori"]
    B --> C["Latents Initialization<br/>Inject Gaussian Noise into Hole Masks + Anchor Background Latents"]
    C --> D["Denoising Timestep Iteration & UNet Forward Pass"]
    D --> E["Content-aware Attention<br/>Multiplicative Scaling and Additive Bias for Region-Selective Focus"]
    E --> F["Multi-Object Energy Guidance<br/>Spatially Disentangled Inpainting, Relocation, and Background Energy Updates"]
    F --> G["Joint Latent Gradient Step and Final Latent Decoding"]

Key Designs¶

1. Latents Initialization: Eradicating Inpainting Ghosts via Selective Noise Injection When manipulating images starting from DDIM-inverted latents $z_T$, the vacated areas $m_n^{\text{ipt}}$ retain strong structural and chromatic imprints of the original objects. Standard diffusion guidance struggles to overcome this inherited prior, frequently producing residual shadows and incomplete erasures. SIMON introduces an elegant initialization strategy: at the beginning of the reverse trajectory, the latent codes within inpainting regions $m_n^{\text{ipt}}$ are explicitly overwritten with random Gaussian noise sampled from a standard normal distribution: $z_T[m_n^{\text{ipt}}] \sim \mathcal{N}(0, \mathbf{I})$. Simultaneously, to prevent background drift, the untouched background region $m^{\text{bg}}$ is locked to the original cached inverted latent trajectory during the first $k$ denoising steps: $z_t^{\text{gen}}[m^{\text{bg}}] = z_t^{\text{ori}}[m^{\text{bg}}]$ (for $t < k$). This dual mechanism purges residual ghost artifacts while harnessing undisturbed structural background context to guide seamless generative hole filling.

2. Content-aware Attention: Region-Adaptive Multiplicative and Additive Attention Modulation Standard key-value sharing often suffers from attention diffusion when multiple semantically similar objects are present in the same scene. To focus attention strictly within instance-relevant corridors, SIMON formulates a content-aware attention modulation function: $$f(\gamma, \delta, A_t(Q_t^{\text{gen}}, K_t^{\text{ori}})) = \gamma \cdot A_t(Q_t^{\text{gen}}, K_t^{\text{ori}}) + \delta$$ where multiplicative factor $\gamma > 1$ amplifies existing salient query-key correlations, and additive offset $\delta > 0$ provides direct linear amplification when baseline attention is weak or dispersed. The self-attention matrix is partitioned into four distinct regional scenarios: - Relocation pairs: when $Q \in \hat{m}_n$ and $K \in m_n$, parameters $(\gamma_1, \delta_1)$ reinforce feature transfer from original to target locations; - Inpainting pairs: when $Q \in m_n^{\text{ipt}}$ and $K \in m_n^{\text{ref}}$, parameters $(\gamma_2, \delta_2)$ draw context from reference background patches; - Background preservation pairs: when $Q \in m^{\text{bg}}$ and $K \in m^{\text{bg}}$, parameters $(\gamma_3, \delta_3)$ enforce background self-consistency; - Cross-region interactions remain unamplified. This spatial decomposition eliminates multi-instance attention crosstalk.

3. Multi-Object Energy Guidance: Spatially Disentangled Tripartite Energy Optimization To bypass the latency explosion and cumulative error of sequential object edits, SIMON formulates a composite energy objective $\mathbf{E}$ that unifies all subtasks. Using a feature similarity metric $g(a, b)$, three subtask-specific terms evaluate correspondence between intermediate generation features $F_t^{\text{gen}}$ and cached inversion features $F_t^{\text{ori}}$: $$\mathbf{E}_{\text{ipt}} = \sum_{n=1}^N \frac{1}{\alpha + \beta \cdot g(F_t^{\text{gen}}[m_n^{\text{ipt}}], F_t^{\text{ori}}[m_n^{\text{ref}}])}$$ $$\mathbf{E}_{\text{relo}} = \sum_{n=1}^N \frac{1}{\alpha + \beta \cdot g(F_t^{\text{gen}}[\hat{m}_n], F_t^{\text{ori}}[m_n])}$$ $$\mathbf{E}_{\text{bg}} = \frac{1}{\alpha + \beta \cdot g(F_t^{\text{gen}}[m^{\text{bg}}], F_t^{\text{ori}}[m^{\text{bg}}])}$$ To prevent spatial gradient interference, the composite energy function integrates spatial mask weighting: $$\mathbf{E} = m^{\text{bg}} \cdot w_b \mathbf{E}_{\text{bg}} + (1 - m^{\text{bg}}) \cdot (w_r \mathbf{E}_{\text{relo}} + \mathbf{E}_{\text{ipt}})$$ At each sampling step $t$, the intermediate latent is updated via energy gradient descent: $z_{t-1} = \hat{z}_t^{\text{gen}} - \eta \nabla_{z_t^{\text{gen}}} \mathbf{E}$. This formulation enables all relocated objects and inpainting zones to converge synchronously in a single forward generation pass, keeping computational runtime constant regardless of object quantity.

Key Experimental Results¶

Main Results¶

Experiments are conducted on an annotated 3D-FUTURE benchmark comprising 1,000 images with 3,405 annotated instances (2 to 5 objects per image). Evaluation spans background consistency (MSE, LPIPS), navigation consistency (MSE, LPIPS), instance identity preservation (DINOv2), and global image quality (FID). SIMON is evaluated against training-free baselines (DragonDiff, DiffEditor, GeoDiffuser) and supervised counterparts (MagicFixup pre-trained on 5M images, Insert+LaMA trained on 159K pairs).

Method	Background MSE ($\times 10^3$)↓	Background LPIPS ($\times 10^{-1}$)↓	Navigation MSE ($\times 10^3$)↓	Navigation LPIPS ($\times 10^{-1}$)↓	Identity DINO↑	Image FID↓
MagicFixup (Supervised 5M)	0.340	1.422	1.337	4.925	0.772	21.964
Insert+LaMA (Two-stage)	0.216	1.523	2.723	7.661	0.867	30.238
DragonDiff (Training-free)	0.539	2.163	1.906	5.543	0.836	30.977
DiffEditor (Training-free)	0.384	1.856	1.721	5.305	0.849	30.193
GeoDiffuser (Training-free)	0.649	2.353	2.237	7.120	0.797	30.633
SIMON (Ours)	0.209	0.969	1.649	4.531	0.869	22.826

SIMON dramatically surpasses prior training-free methods across all evaluation axes: compared to DiffEditor, SIMON reduces background MSE by 45.6%, lowers background LPIPS by 47.8%, improves navigation LPIPS by 14.6%, and improves FID from 30.193 to 22.826, while consuming only 30.3% of DiffEditor's inference runtime. In addition, when objects scale from 2 to 5, runtime for sequential methods climbs from ~60s to >180s, whereas SIMON maintains a flat, efficient runtime curve.

Ablation Study¶

Table 2 and Table 3 isolate the individual contributions of Energy Guidance (EG), Content-aware Attention (CA), and Latents Initialization (LI) across multi-object navigation and isolated inpainting subtasks. Effective inpainting requires high residual suppression (large MSE/LPIPS vs. edited input) and low reconstruction fidelity error (small MSE/LPIPS vs. pristine ground truth).

Configuration / Variant	Subtask / Setting	Metric 1	Metric 2	Overall FID↓	Note
Baseline (DDIM+KV)	Multi-Object Navigation	Background LPIPS: 1.757	Navigation LPIPS: 5.229	28.403	Naive attention sharing with mutual interference
+EG	Multi-Object Navigation	Background LPIPS: 1.017	Navigation LPIPS: 4.752	23.212	Joint energy guidance yields massive perceptual gains
+EG+CA	Multi-Object Navigation	Background LPIPS: 1.008	Navigation LPIPS: 4.525	22.786	Focused attention modulation optimizes relocation fidelity
SIMON Full (+EG+CA+LI)	Multi-Object Navigation	Background LPIPS: 0.969	Navigation LPIPS: 4.531	22.826	Full model achieves peak background consistency
Inpainting: SIMON Full	Object Removal & Inpainting	Residual Supp. MSE: 6.106↑	Reconst. Fid. MSE: 1.964↓	-	Optimal balance of artifact removal and coherence
Inpainting: w/o LI	Object Removal & Inpainting	Residual Supp. MSE: 3.319↑	Reconst. Fid. MSE: 3.155↓	-	Inverted latent ghosting severely degrades inpainting

Key Findings¶

Energy Guidance anchors structural coherence: Adding multi-object energy guidance (+EG) drops background LPIPS from 1.757 to 1.017 and improves FID by 5.19 points, demonstrating the necessity of joint gradient-based trajectory steering.
Latents Initialization eliminates persistent ghosting: In ablation of the inpainting subtask, removing LI halves residual suppression MSE (from 6.106 to 3.319) and degrades reconstruction MSE from 1.964 to 3.155, confirming that overwriting inverted latents with Gaussian noise is vital to prevent ghosting remnants.
Content-aware Attention resolves instance confusion: Modulating Query-Key weights concentrates attention strictly on target instances rather than spreading across co-occurring identical categories, pushing navigation LPIPS to its best score of 4.525.

Highlights & Insights¶

Spatially disentangled multi-energy field: Decomposing multi-object editing into three concurrent, mask-bounded sub-energies enables gradient-based optimization within a single reverse diffusion pass, reducing computational complexity from $O(N)$ sequential passes to $O(1)$ parallel execution.
Training-free pipeline rivaling supervised models: Without fine-tuning on millions of paired samples, SIMON beats the heavily supervised MagicFixup in background preservation (LPIPS 0.969 vs 1.422) and identity retention (DINO 0.869 vs 0.772).
Seamless extension to Diffusion Transformers: The spatial attention re-weighting and latent feature similarity mechanisms naturally generalize to token-grid architectures like Flux without architectural retraining.

Limitations & Future Work¶

Global lighting and shadow inconsistency: Because editing guidance is confined strictly within 2D binary instance masks, external contact shadows and cast reflections (e.g., reflections on water surfaces) are not automatically updated, occasionally producing visual discrepancies with environmental illumination.
Complex depth and occlusion reasoning: In scenes where relocated objects intersect or experience complex depth layering, 2D planar masks lack 3D occlusion awareness, which can produce unnatural boundaries.
Future directions: Integrating differentiable lighting transport models and physical depth priors into the composite energy formulation presents a promising pathway for illumination-harmonized multi-object 3D manipulation.

vs DragonDiffusion & DiffEditor: Prior methods apply single-object energy guidance sequentially, causing severe latency degradation ($3\times$ longer runtime) and error accumulation during complex object shuffling; SIMON updates all instances concurrently via a unified spatial energy field.
vs GeoDiffuser: GeoDiffuser enforces geometric attention constraints but suffers from severe inpainting distortions (reconstruction MSE of 6.524); SIMON's latent initialization technique cuts reconstruction error down to 1.964 while preserving background fidelity.
vs MagicFixup: MagicFixup requires extensive pre-training across 5 million video/image pairs; SIMON operates completely training-free while delivering higher instance identity fidelity and background coherence.

Rating¶

Novelty: ⭐⭐⭐⭐☆ First training-free framework for synchronized multi-object navigation, introducing region-adaptive attention scaling and composite energy guidance.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous benchmark curated on 3D-FUTURE with 1,000 images and 3,405 instances, evaluated across instance-, regional-, and image-level metrics with extensive ablations and user studies.
Writing Quality: ⭐⭐⭐⭐⭐ Clean mathematical formulation, clear task decomposition, and convincing experimental evidence.
Value: ⭐⭐⭐⭐☆ Substantial practical value for interactive graphic editing, interior design layouts, and scalable image compositing workflows.