Jumping the Landing Phase: Noise Variance Matching Enables Accurate Few-Step Inversion¶

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/
Area: Image Generation
Keywords: Diffusion Model Inversion, DDIM Inversion, Few-Step Reconstruction, Noise Variance Matching, Image Editing

TL;DR¶

Addressing the systematic underestimation of noise magnitude in DDIM inversion caused by input temporal mismatch, this paper introduces Noise Variance Matching (NVM), which aligns predicted noise variance with a forward reference to jump the costly landing phase and achieve accurate few-step inversion with minimal UNet calls.

Background & Motivation¶

Diffusion models have achieved remarkable success in text-to-image generation and real-image manipulation. Inversion, which projects a real image back into the diffusion latent space, forms the bedrock of structure-preserving and conditional image editing. Among tuning-free sampling-based inversion methods, DDIM inversion is widely adopted owing to its deterministic trajectory and optimization-free pipeline. However, DDIM inversion frequently suffers from pronounced reconstruction errors and structural blurring under few-step settings. This limitation arises from an inherent temporal asymmetry in the inversion process: while the forward diffusion process estimates noise from the noisy latent $z_t$, the inversion step must approximate $z_t$ starting from the less noisy latent $z_{t-1}$. Because $z_{t-1}$ features a strictly higher Signal-to-Noise Ratio (SNR), denoisers systematically underestimate the required perturbation magnitude. This mismatch accumulates across timesteps as an under-noising or partial denoising bias, severely degrading latent trajectory fidelity.

To alleviate trajectory deviation, iterative refinement approaches such as ReNoise apply repeated denoising evaluations and post-hoc KL alignment at each timestep. Although iterative refinement substantially improves reconstruction quality, invoking multiple UNet passes per step introduces massive computational overhead, undermining the efficiency of fast sampling. By examining the statistical dynamics of ReNoise, the authors uncover that the noise refinement trajectory naturally bifurcates into two distinct phases: a rapid "landing phase" where noise variance and mean undergo substantial adjustments, and a subsequent "steady phase" where statistical metrics stabilize. Most perceptual fidelity gains occur within this short landing phase, which also accounts for the majority of the computational cost, while additional iterations in the steady phase yield diminishing returns.

This key observation leads to an incisive question: can we directly correct the noise variance underestimation at the very beginning of each step, thereby bypassing the costly iterative landing phase altogether? Core idea: by explicitly matching the predicted noise variance to that of an uncorrupted forward reference sample (NVM), the latent state is warm-started directly into the steady phase, enabling high-fidelity few-step diffusion inversion at a fraction of the computational budget.

Method¶

Overall Architecture¶

The proposed Noise Variance Matching (NVM) framework operates across each discrete inversion timestep $t$ via a structured three-stage pipeline: variance alignment, warm-start update, and lightweight steady refinement. Taking the original image latent $z_0$ and total inversion steps $T$ as input, the pipeline first constructs a forward reference latent $z_t^*$ using standard Gaussian perturbation to measure the target noise variance under the denoiser. It then compares this reference variance against the initial prediction from $z_{t-1}$ to compute an analytic scaling factor $\Delta_{\text{var}}$. Guided by an attention-derived spatial mask $M$, the variance correction is applied selectively to salient regions, boosting the underestimated noise magnitude straight into the steady-state regime. Finally, a deterministic warm-start step produces an initial latent $z_{\text{init}}$, followed by only 1 to 2 lightweight refinement iterations to eliminate sub-pixel residual errors and secure the final inverted latent $z_T$.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input latent z_t-1 and original clean latent z_0"] --> B["Forward Reference Sampling and Variance Estimation"]
    B --> C["Noise Variance Matching<br/>Compute global variance scaling factor Δvar"]
    C --> D["Attention-Guided Spatial Mask<br/>Fuse cross/self-attention for salient mask M"]
    D --> E["Steady-State Warm-Start Lightweight Refinement<br/>One-step warm start and shallow K=2 refinement"]
    E --> F["Output precise latent z_t and propagate to terminal z_T"]

Key Designs¶

1. Noise Variance Matching: Explicit Alignment with Forward Reference to Jump the Landing Phase To address the fundamental flaw of DDIM inversion—where predicting noise from $z_{t-1}$ systematically underestimates noise magnitude—NVM directly eliminates the multi-pass landing phase. Analytically, the inversion step error is formulated as: $$z_t^{\text{inv}} - z_t^{\text{true}} = \eta_t [\epsilon_\theta(z_{t-1}, t) - \epsilon_\theta(z_t, t)]$$ where $\eta_t := \sigma_t - \frac{\alpha_t \sigma_{t-1}}{\alpha_{t-1}} > 0$. Because $\sigma_{t-1} < \sigma_t$, the denoiser encounters an input with higher-than-expected SNR, keeping the noise direction largely consistent while shrinking its magnitude: $\epsilon_\theta(z_{t-1}, t) \approx \rho \epsilon_\theta(z_t, t)$ with $\rho \in (0, 1)$. This injects an implicit negative noise term $\eta_t(\rho - 1)\epsilon_\theta(z_t, t)$, causing an artificial partial denoising effect. NVM remedies this by synthesizing an uncorrupted forward reference latent $z_t^* = \alpha_t z_0 + \sigma_t \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, evaluating its denoiser response, and deriving a variance correction factor: $$\Delta_{\text{var}} = 1 + \frac{|\mathrm{Var}(\epsilon_\theta(z_t^*, t)) - \mathrm{Var}(\epsilon_\theta(z_{t-1}, t))|}{\gamma}$$ where $\gamma = 6$ governs the adjustment strength. Scaling the initial prediction by $\epsilon'_\theta = \epsilon_\theta(z_{t-1}, t) \cdot \Delta_{\text{var}}$ instantly bridges the variance gap that otherwise requires multiple iterative loops in ReNoise, completing the landing transition in a single step.

2. Attention-Guided Spatial Mask: Semantic and Texture-Adaptive Localized Modulation To prevent global variance scaling from over-sharpening smooth background regions (such as clear skies or homogeneous textures) and introducing unnatural artifacts, this design introduces a spatial guidance mask. Internal attention layers naturally partition semantic and geometric features: low-resolution cross-attention maps concentrate on core semantic concepts, whereas high-resolution self-attention maps delineate fine spatial boundaries. NVM combines both attention signals and thresholds them with a Top-K selection ratio ($\gamma_{\text{scope}} = 0.25$) to generate a binary guidance mask $M \in \{0, 1\}^{1 \times 1 \times H \times W}$. In practice, $M$ is extracted using a fixed auxiliary prompt token (SDXL token index 3), remaining completely independent of user-specified text prompts. The masked noise adjustment is formulated as: $$\epsilon'_\theta = \epsilon_\theta(z_{t-1}, t) \odot [M \odot (\Delta_{\text{var}} - 1) + 1]$$ This formulation confines the per-channel scaling factor $\Delta_{\text{var}}$ strictly to salient high-frequency structures ($M=1$), while leaving smooth low-frequency backgrounds unperturbed, thereby preserving overall perceptual coherence.

3. Steady-State Warm-Start Lightweight Refinement: High-Fidelity Residual Correction with Minimal Budget To avoid the computational waste of blind cold-start iterations (e.g., ReNoise's default $K=9$), this design anchors optimization directly in the steady-state basin. With the variance-calibrated noise $\epsilon'_\theta$, the algorithm executes an initial warm-start inversion step: $z_{\text{init}} = \text{InvStep}(z_{t-1}, \epsilon'_\theta)$. Because the major magnitude discrepancy has been resolved by NVM, $z_{\text{init}}$ already resides within the steady-phase convergence regime. From this optimized starting point, the framework requires only $K=2$ shallow refinement passes and lightweight distribution regularization to eliminate fine-grained spatial residuals. This functional division—where NVM establishes structural scale and shallow refinement polishes pixel details—slashes the per-step UNet computation by more than two-thirds while maintaining superior inversion fidelity.

Key Experimental Results¶

Main Results¶

The authors evaluated few-step image reconstruction fidelity under a 5-step denoising budget on the HPDv2 and MS-COCO benchmarks using SDXL.

Dataset	Method	CLIP-I ↑	MSE ↓	PSNR ↑	LPIPS ↓
HPDv2	DDIM	67.18	0.0159	18.54	0.4687
HPDv2	FreeInv	83.58	0.0102	20.46	0.2902
HPDv2	ReNoise	88.97	0.0068	22.62	0.2505
HPDv2	EasyInv	80.92	0.0148	18.89	0.4015
HPDv2	BDIA	76.00	0.0169	17.91	0.6226
HPDv2	BELM	53.56	0.1774	7.57	0.8577
HPDv2	EDICT	90.41	0.0064	22.23	0.3195
HPDv2	Ours	93.55	0.0040	24.73	0.1676
MS-COCO	DDIM	73.76	0.0127	19.59	0.3721
MS-COCO	FreeInv	79.59	0.0118	19.86	0.3087
MS-COCO	ReNoise	80.32	0.0419	19.80	0.3270
MS-COCO	EasyInv	74.03	0.0124	19.67	0.3684
MS-COCO	BDIA	69.05	0.0204	17.12	0.6252
MS-COCO	BELM	51.77	0.1795	7.55	0.8403
MS-COCO	EDICT	85.67	0.0090	20.83	0.3560
MS-COCO	Ours	90.27	0.0058	23.30	0.1923

On the PIE image editing benchmark, NVM also achieves the best trade-off between background preservation and text controllability: it secures the lowest background LPIPS (0.1406) and highest SSIM (0.8526), while outperforming all optimization-free baselines in whole-image CLIP-T (32.55) and edited-region CLIP-T (28.10).

Ablation Study¶

The ablation study on HPDv2 systematically isolates the contributions of Global Scaling (G.S.), Noise Variance Matching (NVM), Spatial Masking, and refinement steps (R1 / R2):

Config	Global Scale (G.S.)	NVM	Spatial Mask	Refinement	CLIP-I ↑	MSE ↓	PSNR ↑	LPIPS ↓	Note
Baseline	✗	✗	✗	0	67.18	0.0159	18.54	0.4687	Vanilla DDIM, severe under-noising
+ Fixed Scale	✓ (1.01)	✗	✗	0	77.45	0.0113	20.15	0.3305	Heuristic global magnitude scaling
+ Dynamic NVM	✗	✓	✗	0	84.64	0.0101	20.53	0.2159	Bypasses landing phase via variance alignment
+ Spatial Mask	✗	✓	✓	0	84.48	0.0092	21.04	0.2692	Prevents over-sharpening in smooth areas
+ 1 Refinement	✗	✓	✓	R1 (K=1)	88.81	0.0060	23.01	0.2101	Shallow steady-state residual calibration
Full Model	✗	✓	✓	R2 (K=2)	93.55	0.0040	24.73	0.1676	Full model: structural fidelity + pixel accuracy

Key Findings¶

Variance alignment is the decisive bottleneck for few-step inversion: simply scaling noise by a fixed 1.01 factor increases CLIP-I by +10.27 points, while dynamically matching variance via NVM yields a +17.46 gain and slashes LPIPS from 0.4687 to 0.2159, confirming that noise magnitude underestimation drives trajectory deviation.
Spatial masking delivers complementary frequency-domain benefits: while global NVM dramatically recovers structural semantics, localized masking protects low-frequency areas, boosting PSNR by +0.51 dB (from 20.53 to 21.04) and reducing MSE to 0.0092.
Dramatic efficiency gains: NVM (Full) requires only 45 total UNet evaluations (14.24s runtime), delivering over a 3× speedup compared to ReNoise's 93 calls (45.87s), while surpassing all baselines across every reconstruction metric.

Highlights & Insights¶

Conceptualizing iterative inversion as landing vs. steady phases: the authors identify that ReNoise's heavy compute is consumed by an early landing phase adjusting mean and variance, making upfront alignment an elegant shortcut.
Training-free forward reference matching: synthesizing reference noise statistics from $z_0$ provides a sound mathematical anchor without requiring offline training or test-time optimization.
Cross-paradigm applicability to Flow Matching: the core insight generalizes beyond diffusion variance modulation to continuous velocity field alignment in Flow Matching models like FLUX.

Limitations & Future Work¶

Reliance on clean latent for forward reference: constructing the reference sample requires access to the uncorrupted input latent $z_0$, which may constrain cascaded super-resolution pipelines.
Heuristic thresholding for attention masking: the spatial mask depends on a fixed Top-K selection ratio ($\gamma_{\text{scope}} = 0.25$), which may require adaptive tuning on highly crowded visual scenes.
Future directions: developing channel-adaptive variance modulation and extending statistical alignment to video diffusion and 3D generation.

vs DDIM Inversion: DDIM inversion estimates noise from lower-noise latents $z_{t-1}$, incurring severe under-noising bias; NVM eliminates this systematic error via variance matching.
vs ReNoise: ReNoise expends 9 iterative UNet passes searching for stable noise statistics; NVM warm-starts directly into the steady phase, achieving higher fidelity with only 2 refinement steps and 3× speedup.
vs EDICT / BDIA: Exact inversion methods use coupled ODE trajectories that require doubled memory or rigid computation graphs; NVM retains single-stream simplicity while outperforming them in few-step fidelity and editability.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Discovers the two-phase convergence mechanism of iterative inversion and proposes variance matching to skip the landing phase]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensively validated across SDXL, SD1.5, LCM, SDXL Turbo, and FLUX with rigorous ablations and editing benchmarks]
Writing Quality: ⭐⭐⭐⭐⭐ [Mathematical error derivation is elegant, accompanied by clear visual evidence and intuitive phase breakdowns]
Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly efficient solution for real-time few-step diffusion inversion and editing]