SONIC: Spectral Optimization of Noise for Inpainting with Consistency

Conference: ECCV 2026
arXiv: 2511.19985
Code: https://ubc-vision.github.io/sonic/ (Project Page)
Area: Diffusion Models / Image Generation
Keywords: Image Inpainting, Initial Noise Optimization, Spectral Preconditioning, Training-Free, Linear Approximation

TL;DR

SONIC proposes a training-free diffusion-based image inpainting method. The core idea is to optimize the initial noise map so that the reconstructed image after denoising faithfully restores unmasked regions. It bypasses expensive backpropagation via denoising trajectory linearization and uses Adam in the spectral domain for preconditioned optimization to achieve stable convergence, outperforming existing methods (including the specially trained BrushNet) across three standard inpainting datasets.

Background & Motivation

Diffusion and flow models have become the mainstream solutions for inverse problems such as image inpainting. Existing approaches are divided into two categories: training dedicated inpainting models (e.g., BrushNet), which yield good results but are expensive and limited in generalization, and training-free methods (e.g., BLD, FLAIR, FlowChef) that leverage guidance or posterior sampling from pre-trained models. The latter are flexible but significantly underperform compared to dedicated models, with inpainting results often being structurally inconsistent with the original image.

The key challenge lies in the fact that the initial noise map plays a decisive role in the final generation structure (prior work has demonstrated that different initial noise maps lead to entirely different scene layouts and inpainting outcomes). However, optimizing the initial noise requires backpropagating through the entire \(T\)-step denoising chain—which is computationally and memory-wise infeasible for modern large models like SD3.5 (backpropagation of more than one step leads to OOM even on an RTX 5090). Existing works either ignore the optimization of initial noise entirely or leverage tedious hyperparameter tuning or auxiliary networks to bypass it, lacking a practical and stable solution.

The core idea of this paper is to make the initial noise optimizable by linearizing the denoising trajectory (without backpropagating through the denoiser), combined with spectral-domain preconditioning to guarantee optimization stability. This combination makes "optimizing initial noise for inpainting" practically viable for the first time, outperforming dedicated models.

Method

Overall Architecture

The goal of SONIC is to find an optimal initial noise \(x_T^*\) without altering the denoising pipeline itself, such that after \(T\)-step denoising \(\mathcal{D}_T(\cdot)\), the unmasked regions are consistent with the observed image \(y\). The whole process consists of three steps: (1) fill the masked image using nearest-neighbor interpolation and encode it into the latent space via a VAE to obtain the observation \(y\); (2) optimize the initial noise \(X_T\) (the Fourier transform of \(x_T\)) in the spectral domain using a linearized loss and the Adam optimizer for approximately 20 iterations; (3) use the optimized \(x_T = \mathcal{F}^{-1}(X_T)\) as the initial noise and feed it into a standard training-free inpainting pipeline (such as BLD-SD3.5) to complete the inpainting.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Masked Image + Mask"] --> B["Nearest-Neighbor Filling + VAE Encoding<br/>to Obtain Latent Observation y"]
    B --> C["Random Initialization of X_T (Spectral Domain)"]
    C --> D["Denoising Trajectory Linearization<br/>Compute D_T(x_T) and stop-grad"]
    D --> E["Spectral Preconditioned Optimization<br/>Adam Updates X_T in Spectral Domain"]
    E -->|"Iterate ~20 steps"| D
    E --> F["F^{-1}(X_T) to Obtain Optimized x_T"]
    F --> G["BLD Standard Inpainting Pipeline<br/>Output Final Inpainted Image"]

Key Designs

1. Denoising Trajectory Linearization: Practical Optimization Bypassing Backpropagation

A naive approach to optimize the initial noise is to perform gradient descent on the loss \(\mathcal{L} = \|y - A\mathcal{D}_T(x_T)\|^2\), which requires backpropagation through the entire \(T\)-step denoising chain—infeasible even for a single step on large models like SD3.5. A key insight of this paper is that modern flow models (e.g., Rectified Flow used in SD3.5) are trained to render the denoising trajectory as linear as possible. Thus, the entire trajectory can be approximated as a straight line from \(x_T\) to \(\mathcal{D}_T(x_T)\):

\[\hat{x}(t) = [\mathcal{D}_T(x_T) - x_T]_{\text{sg}} \cdot (1 - \frac{t}{T}) + x_T\]

Where \([\cdot]_{\text{sg}}\) is the stop-gradient operator, treating \(\mathcal{D}_T(x_T) - x_T\) as a constant. Substituting \(t=0\) yields the approximated final output \(\hat{x}(0) = [\mathcal{D}_T(x_T) - x_T]_{\text{sg}} + x_T\). The loss function simplifies to:

\[\mathcal{L}_{\text{linear}} = \| y - A([\mathcal{D}_T(x_T) - x_T]_{\text{sg}} + x_T) \|_2^2\]

Since the stop-gradient cuts off the gradient flow through \(\mathcal{D}_T\), this loss is differentiable with respect to \(x_T\) without requiring backpropagation through the denoiser—essentially acting as a Straight-Through Estimator that drops the Jacobian matrix. Through visualization of the cosine similarity between the predicted velocity at each timestep and the average velocity of SD3.5, the supplementary material validates that the trajectory is indeed nearly linear across most timesteps (with only slight deviations in the final steps when the noise is close to zero), providing empirical support for the rationality of this approximation.

2. Spectral-Domain Preconditioned Optimization: Allowing Each Frequency to Converge at Its Own Pace

Direct optimization in the spatial domain using Eq. (3) yields regional instabilities—the denoising results flicker across iterations, and different spatial frequencies converge at different rates. This occurs due to the highly imbalanced spectrum of the latent-space gradient: low-frequency components have large gradient variance, while high-frequency components have small variance. A global learning rate in the spatial domain cannot balance both. Inspired by recent findings that "different frequencies require different strengths during guidance," this work shifts the optimization to the spectral (Fourier) domain: setting \(x_T = \mathcal{F}^{-1}(X_T)\), and optimizing the complex spectral representation \(X_T\) instead of \(x_T\). Crucially, using the Adam optimizer—which independently calculates second-order moments for each frequency component and performs element-wise division in the spectral domain—naturally achieves a preconditioning effect where each frequency has an adaptive learning rate:

\[\mathcal{L}_{\text{spectral}} = \| y - A([\mathcal{D}_T(x_T) - \mathcal{F}^{-1}(X_T)]_{\text{sg}} + \mathcal{F}^{-1}(X_T)) \|_2^2\]

Here lies an important theoretical guarantee: the Fourier transform is a unitary transform, meaning spatial-domain and spectral-domain SGD are mathematically equivalent (\(\Delta x = -\eta g\) gives the same effective update on both ends). However, the \(\text{sgn}(\cdot)\) nonlinearity in Adam does not commute with the DFT matrix (\(\mathcal{F}^{-1} \cdot \text{sgn}(\mathcal{F} \cdot g) \neq c \cdot \text{sgn}(g)\)). Consequently, spectral-domain Adam traces a unique optimization trajectory that spatial-domain Adam can never reproduce simply by scaling up the learning rate. Theoretical analysis in the supplementary material further points out that a block update in the spectral domain corresponds to a globally dense, correlated spatial-domain update—propagating global structural information in a single step, explaining why only 20 optimization steps suffice.

3. Spatial Constraints: Preventing Optimization from Deviating from the Noise Manifold

Despite being stable, spectral-domain optimization faces a pitfall: since Fourier basis functions are globally supported, updates to a single frequency component affect pixels across all spatial locations. For masked regions—pixels with no observational data constraints—the latent variables might gradually drift away from the standard Gaussian distribution \(\mathcal{N}(0,1)\), which the denoising model expects the initial noise to follow. Such a deviation leads to visible color shifts, artifacts, and footprint traces along the mask boundaries in the denoising results.

The solution is simple yet effective: apply a spatial mask to the gradient updates, allowing gradients to flow back only through unmasked (observed) pixel regions while freezing the latent variables of the masked regions (which already follow \(\mathcal{N}(0,1)\) and need no modification). This confines the optimization within an "acceptable initial noise manifold." Experiments demonstrate that removing this constraint causes the FID to spike from 12.88 to 33.27 (on FFHQ), with qualitative results showing severe boundary artifacts and color distortions.

A Complete Example

Taking FFHQ face inpainting as an example: the input is a \(1024 \times 1024\) face photo, with approximately the left half occluded by a rectangular mask. First, nearest-neighbor color filling is applied to the masked region (pixels on the mask boundary are directly copied inward), and the entire image is encoded into the latent space via SD3.5's VAE to obtain the observation \(y\). Then, \(x_T\) is randomly initialized from \(\mathcal{N}(0,1)\), and its Fourier transform yields \(X_T\) as the optimizable parameter. Entering the iterations: in each round, \(X_T\) is inversely transformed back to \(x_T\), a full \(T=20\) step denoising is run to obtain \(\mathcal{D}_T(x_T)\), and the MSE loss (Eq. 4) acting solely on the unmasked region is computed. The Adam optimizer (\(lr=3.0\)) updates \(X_T\) in the spectral domain. By iteration 5, the denoising result already outlines the facial contour; by step 10, the facial features and lighting are roughly aligned; by step 20, the unmasked region is reconstructed almost perfectly. Finally, the optimized \(x_T\) is sent to BLD-SD3.5, which completes the inpainting of the masked region using standard latent blending during denoising. Because the initial noise has locked in the global structure, the final inpainting is highly consistent with the unmasked region in terms of lighting, tone, and geometry.

Loss & Training

The final optimization objective is \(\mathcal{L}_{\text{spectral}}\) in Eq. (4), which is essentially a weighted MSE: the \(L_2\) distance between the denoising output and the observation is computed only on unmasked pixels. The optimizer is Adam (\(\beta_1=0.9, \beta_2=0.999\)), with a learning rate of 3.0 (much higher than spatial-domain Adam's 0.05, since spectral preconditioning allows for larger learning rates). The number of denoising steps is \(T=20\), and the classifier-free guidance scale is 2.0 (consistent with FLAIR). All operations are conducted at a resolution of \(1024 \times 1024\), matching the expected input size of SD3.5. Once optimization is complete, \(x_T\) is passed to BLD-SD3.5 for final inpainting, which does not alter the denoising process itself. This entire pipeline requires no training, no fine-tuning, and no meticulous learning rate tuning, taking approximately 67 seconds per image on an RTX 5090.

Key Experimental Results

Main Results

The evaluation is conducted on three datasets: FFHQ (1000 faces with large rectangular masks), DIV2K (800 natural images with 6 random rectangular masks), and BrushBench (600 images with human-annotated segmentation masks). The following table extracts the three most representative metrics (SSIM/LPIPS/FID); the complete set of 8 metrics can be found in the original paper.

Dataset	Method	SSIM↑	LPIPS↓	FID↓
FFHQ	BLD-SD3.5	0.824	0.180	25.84
FFHQ	FLAIR (400 NFE)	0.829	0.292	19.87
FFHQ	BrushNet	0.759	0.237	17.95
FFHQ	SONIC (Ours)	0.857	0.121	12.88
DIV2K	BLD-SD3.5	0.789	0.144	25.33
DIV2K	FLAIR (400 NFE)	0.768	0.294	22.15
DIV2K	BrushNet	0.575	0.308	29.00
DIV2K	SONIC (Ours)	0.803	0.115	17.70
BrushBench	BLD-SD3.5	0.854	0.169	51.66
BrushBench	FLAIR (400 NFE)	0.855	0.190	51.54
BrushBench	BrushNet	0.740	0.199	50.08
BrushBench	SONIC (Ours)	0.861	0.153	48.07

SONIC achieves optimal performance on SSIM, LPIPS, and FID across all three datasets. Its FID advantage is particularly prominent: 12.88 on FFHQ (compared to the second-best BrushNet's 17.95) and 17.70 on DIV2K (compared to the second-best FLAIR's 22.15). While SONIC is not always the highest on PSNR and CLIP Score, the paper notes that PSNR is not a reliable metric in inpainting tasks (as blurry patches can yield high scores), and CLIP Score only measures semantic alignment with the text prompt without considering structural consistency with unmasked regions. For human preference scores (HPS v2, IR), SONIC is similarly leading or close to optimal.

User study: In a single-blind 2AFC test with 37 participants (1665 pair-wise comparisons), SONIC achieved an overall win rate of 90.0% (95% CI [88.4, 91.4]), with a 94.0% win rate against BLD-SD3.5, 80.5% against FLAIR, and 79.9% against BrushNet. All comparisons are statistically significant.

Ablation Study

Ablation studies on the three core designs performed on FFHQ yield the following results.

Variant	SSIM↑	LPIPS↓	FID↓	IR↑
Full SONIC	0.857	0.121	12.88	0.280
w/o Spectral Preconditioning (Spatial-domain Adam, lr=0.05)	0.853	0.127	14.58	0.216
w/o Spatial Constraints (No gradient freezing on masked regions)	0.833	0.165	33.27	-0.259
Replacing nearest-neighbor filling with GT original image (Theoretical upper bound)	0.856	0.121	12.77	0.280

Three conclusions: (1) Removing spatial constraints is the most detrimental—FID surges from 12.88 to 33.27, and IR even turns negative, validating the vital importance of keeping the noise within the valid manifold. (2) Although the drop in metrics without spectral preconditioning is less drastic than without spatial constraints, IR decreases from 0.280 to 0.216, reflecting decayed human-perceived quality. This aligns with the qualitative observation that inpainting results become unstable or fail outright when spectral preconditioning is disabled. (3) Replacing nearest-neighbor filling with the GT original image yields almost no extra benefit (FID 12.77 vs. 12.88), illustrating that simple nearest-neighbor filling is sufficient, and SONIC's robustness obviates the need for complex latent-space filling strategies.

Ablation on optimization iterations (FFHQ): 10 rounds (200 NFE) yield FID 12.78, 20 rounds (400 NFE) yield FID 12.88, and 30 rounds (600 NFE) yield FID 12.84—returns diminish after 20 rounds, hence 20 rounds is chosen as the optimal trade-off in the paper.

Key Findings

• Spatial constraints are the cornerstone of performance: Removing them deteriorates FID by nearly 3x, far exceeding other ablations. This demonstrates that "making the optimized noise still look like noise" is more fundamental than "how to optimize it."
• The greatest value of spectral preconditioning lies in stability rather than peak metrics: Although the absolute gaps in SSIM/LPIPS/FID are narrow, spatial-domain Adam requires meticulous learning rate tuning (0.05 vs. 3.0 in the spectral domain) and suffers from catastrophic failures on certain samples—where the inpainting results completely collapse—whereas spectral-domain optimization remains stable across all samples.
• SONIC is equally effective on non-flow models: In the supplementary material, the authors test on SD1.5 and SDXL (which are diffusion models, not flow models, and do not possess strictly linear trajectories). The initial noise optimized by SONIC still significantly outperforms random noise, yielding a clear boost in generation quality. This indicates that while the linear approximation is based on flow model assumptions, it generalizes well to diffusion models in practice.
• The robustness of nearest-neighbor filling is surprising: This seemingly crude strategy performs almost identically to encoding the GT original image, indicating that SONIC's loss function is highly insensitive to VAE encoding errors—a highly valuable engineering finding.

Highlights & Insights

• The paradigm of "optimizing the initial noise instead of modifying the denoising process" was widely considered infeasible in the diffusion model era. This paper makes it viable again using two simple yet non-trivial techniques. Linearization is an "engineered, good-enough" approximation (with empirical trajectory linearity validation and flow model theoretical guarantees), while spectral preconditioning is a "mathematically non-trivial" refinement (benefited by the non-commutative property of Adam's nonlinearity and DFT). Their combination produces a 1+1>2 effect.
• The theoretical analysis of spectral-domain optimization is noteworthy: In the supplementary material, it is proven that spectral-domain Adam traces an optimization trajectory within limited steps that spatial-domain Adam can never reproduce—because the \(\text{sign}(\cdot)\) nonlinearity does not commute with the Fourier basis. A single-step update in the spectral domain is equivalent to a globally dense, correlated spatial update, which is crucial in such a short 20-step optimization window.
• Nearest-neighbor latent-space filling is a highly reusable engineering trick: Any method that requires solving inverse problems via optimization in the latent space could benefit from this. Instead of designing complex encoding strategies to handle occluded/missing regions, one can simply perform raw filling and rely on the robustness of the subsequent optimization. This design philosophy ("strong back-end optimization allows for crude front-end input") can be transferred to other inverse problem tasks (e.g., super-resolution, deblurring).

Limitations & Future Work

• Limitations acknowledged by the authors: Currently validated only on image inpainting tasks. Although linearization and spectral preconditioning are theoretically generalizable to other inverse problems like super-resolution, deblurring, and video inpainting, they remain empirically unproven.
• High computational overhead: It takes around 67 seconds per image on an RTX 5090. Though comparable to FLAIR with the same NFE, it is far slower than BLD (4 seconds) and BrushNet (5 seconds), making it unusable for real-time or interactive scenarios.
• Lack of analysis on mask size sensitivity: The paper does not systematically study the performance changes of SONIC under different mask ratios. When the mask area is vast (e.g., 90%), the observational feedback provided by unmasked areas is extremely sparse, and the linearized loss might lack sufficient signal.
• Missing theoretical error bounds for the linear approximation: Although shown to be empirically effective, there is no formal analysis regarding the systemic bias introduced by the linear approximation onto the original problem (1) or under what conditions it breaks down.
• Specific refinement ideas: (1) Combine spectral preconditioning with more efficient denoising scheduling (such as fast samplers like DPM-Solver) to reduce NFE requirements; (2) extend linearization/spectral optimization to video inpainting—as the supplementary material already shows preliminary results on the Wan video model; (3) explore fine-tuning an "initial noise proposal network" with a small amount of data to replace iterative optimization and accelerate inference.

Related Work & Insights

• vs FLAIR: FLAIR utilizes variational posterior sampling to simultaneously update the initial noise and the final estimate, representing the strongest baseline among training-free methods. However, FLAIR's optimization of the initial noise is implicit and embedded inside the posterior sampling formulation rather than explicitly targeted. This often results in structural inconsistencies (e.g., blurry patches on DIV2K). SONIC explicitly optimizes the initial noise with the objective of "matching the denoising output with the unmasked regions," yielding significantly better structural consistency.
• vs Concurrent Work on Direct Jacobian Dropping (FlowOpt): A concurrent work, FlowOpt, also drops the vector-Jacobian product to achieve a loss similar to Eq. (3). However, it lacks spectral preconditioning, making the optimization extremely unstable—it requires starting from noise initialized by another inversion method and demands exhaustive learning rate tuning. SONIC's spectral preconditioning resolves this exact issue, allowing optimization to start directly from random Gaussian noise without auxiliary initialization techniques.
• vs BrushNet: BrushNet is a classic representative of training-based methods, which simplifies image inpainting by separating masked image features from noisy latents. Its advantages are fast inference and seamless boundaries, but its disadvantage is poor generalization to unseen mask shapes during training (degenerating into a "storytelling-style" synthesis task under DIV2K's horizontal stripe masks). SONIC demonstrates that "training dedicated models" is not the only path to superior inpainting—training-free methods can completely outperform them across key metrics if initial noise optimization is properly executed.

Rating

• Novelty: ⭐⭐⭐⭐☆ It paves the path of "optimizing initial noise for inpainting" once deemed impractical in the diffusion model era. The combination of linearization and spectral preconditioning introduces fresh perspectives both in engineering and theory, though individual components (such as the Straight-Through Estimator or spectral-domain optimization) are not pioneering concepts on their own.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Supported by three standard datasets, five baseline categories, eight metrics, thorough ablation studies, a user study, and extensive experiments in the supplementary materials (including tests on non-flow models, video models, and additional noise-optimization baselines). The coverage is broad and the findings are highly self-consistent.
• Writing Quality: ⭐⭐⭐⭐☆ Methodological and experimental descriptions are clear, with comprehensive mathematical derivations. The empirical validation of the linearization assumption and the theoretical analysis of spectral optimization in the supplementary materials are strong pluses. One marginal issue is the slightly long-winded justification when PSNR/CLIP metrics trail behind the baselines.
• Value: ⭐⭐⭐⭐☆ Offers a new paradigm for training-free diffusion-based inpainting—focusing on initial noise optimization at the input end rather than altering the denoising steps at the output end. This "input-end optimization" mindset is highly generalizable to other generative inverse problems and holds potentially broad impact.

Related Papers

• [ECCV 2026] Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning
• [ICLR 2026] Diverse Text-to-Image Generation via Contrastive Noise Optimization
• [ECCV 2026] Dual-End Consistency Model
• [ICML 2026] Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs
• [CVPR 2026] From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition