Source-Agnostic Image Translation Based on Latent Aware Adaptive Masking¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/dtoma95/PM-Edit
Area: Image Generation
Keywords: Image-to-Image Translation, Diffusion Models, Adaptive Masking, Source-Agnostic, Progressive Inpainting

TL;DR¶

PM-Edit introduces a source-agnostic unsupervised image-to-image translation framework that uses offline prediction discrepancy statistics on the target domain to build a time-dependent dynamic threshold, progressively growing a minimal mask throughout reverse diffusion and guiding inpainting to achieve high-realism and faithful translation without requiring source-domain training.

Background & Motivation¶

Unsupervised Image-to-Image (I2I) translation aims to map images across visual domains without requiring paired training data. Early approaches predominantly built upon Generative Adversarial Networks (GANs), employing bidirectional cycle consistency (CycleGAN) or patch-wise contrastive learning (CUT) to isolate target-specific features while preserving domain-shared content. However, GAN-based frameworks frequently suffer from mode collapse and limited distribution coverage, often producing unnatural distortion or severe structural artifacts when confronted with complex geometric transformations and high-resolution details.

Denoising Diffusion Probabilistic Models (DDPM/DDIM) have significantly elevated generative fidelity and mode coverage, sparking strong interest in diffusion-driven I2I translation. Modern approaches such as EGSDE, CycleDiffusion, BBDM, and UNSB leverage energy guidance, bidirectional inversion, or direct domain-to-domain bridge processes. Nevertheless, these methods inherently depend on joint modeling or specialized pretraining on both source and target domains, making them rigid in real-world scenarios where source distributions are arbitrary, unseen, or dynamically changing. Conversely, existing source-agnostic techniques like SDEdit and ILVR rely exclusively on a target-domain generator, but they perturb and filter images indiscriminately without distinguishing domain-specific attributes from shared invariant structures, creating an unavoidable compromise between realism and faithfulness.

Mask-guided diffusion approaches such as DiffEdit attempt to preserve invariant regions by identifying discrepancy regions in the latent space. Yet, existing masking methods typically compute a binary mask at a single pre-selected time step using a fixed heuristic threshold and hold it constant across the entire reverse trajectory. In reality, the predictive confidence and semantic granularity of diffusion models fluctuate drastically across noise levels—coarse geometric structures emerge in early stages while high-frequency textures resolve near the end. A static mask inevitably causes either under-editing of global attributes or over-masking of fine details. Core idea: by profiling the empirical distribution of prediction discrepancies on target-domain samples across time steps, we construct a time-dependent statistical threshold that adaptively refines a minimal necessary mask during reverse diffusion, driving progressive inpainting to modify domain-divergent regions without any source-domain training.

Method¶

Overall Architecture¶

PM-Edit operates via a three-phase pipeline: offline profiling of target-domain prediction discrepancy statistics, online dynamic accumulation of minimal binary masks during reverse diffusion, and mask-guided progressive inpainting. The framework is strictly training-free and source-agnostic, requiring only an off-the-shelf DDPM/DDIM backbone \(\epsilon_\theta\) pretrained on the target domain \(\mathcal{X}\).

Given a reference image \(y\) from an arbitrary source distribution, the system computes the empirical mean \(\mu_{D_t}\) and standard deviation \(\sigma_{D_t}\) of single-step prediction discrepancies across target samples in advance, yielding a time-dependent threshold \(\delta_t\). During the online reverse diffusion trajectory from \(T\) down to \(1\), the framework measures the local prediction divergence \(D_t\) between the current synthesized latent state \(x_t\) and the masked noisy reference \(\tilde{y}_t\). Pixels exceeding \(\delta_t\) form the incremental discrepancy mask \(\Delta M_t\), which updates the accumulated mask \(M_t\) via set union. Finally, the sequence of masks \([M_T, \dots, M_1]\), smoothed with slight dilation and Gaussian blurring, guides a RePaint-style progressive inpainting process to synthesize the final translated image \(x_0\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input source image y and pretrained target diffusion backbone"] --> B["Offline Time-Dependent Statistical Thresholding<br/>Target domain single-step discrepancy profiling -> Dynamic threshold curve"]
    B --> C["Latent-Aware Adaptive Mask Accumulation<br/>Reverse trajectory discrepancy evaluation -> Iterative union update of minimal mask"]
    C --> D["Multi-Batch Noise Averaging & Progressive Inpainting<br/>Multi-seed outlier mitigation -> Step-wise dynamic mask RePaint inpainting"]
    D --> E["Output source-agnostic translated image x0"]

Key Designs¶

1. Offline Time-Dependent Statistical Thresholding: Eliminating Noise-Level Discrepancy Bias

Because the reverse diffusion trajectory spans a wide spectrum from isotropic Gaussian noise to clean image manifolds, the network's predictive variance varies dramatically across time steps. In intermediate steps where global semantic geometry is resolved, prediction error variance peaks; near terminal steps, variance sharply contracts. A static threshold either over-masks early noisy states or misses subtle fine-grained deviations. To overcome this, PM-Edit profiles the target distribution \(z \sim \mathcal{X}\) offline: for each noisy pair \(z_t\) and \(z_{t+1}\), the network performs a single-step DDIM projection to obtain \(\hat{z}_t\), and records the empirical pixel-wise mean \(\mu_{D_t}\) and standard deviation \(\sigma_{D_t}\) of the discrepancy \(D_t(z_t, \hat{z}_t)\). The adaptive threshold at step \(t\) is defined as:

\[\delta_t = \mu_{D_t} + \lambda \sigma_{D_t}\]

where \(\lambda\) is a single global scalar hyperparameter. Profiling is performed once per target domain and readily transfers to any unknown source distribution, providing a mathematically grounded boundary that mirrors the model's noise-level confidence.

2. Latent-Aware Adaptive Mask Accumulation: Dynamically Isolating Evolving Cross-Domain Discrepancies

Static single-step masks must adopt conservative boundaries to accommodate subsequent transformations. PM-Edit instead dynamically grows a minimal mask along the reverse diffusion path. At time step \(t\), the current latent state \(x_t\) is sampled via deterministic DDIM from the previously masked latent \(\tilde{x}_{t+1} = M_{t+1} x_{t+1} + (1 - M_{t+1}) y_{t+1}\). Consequently, \(x_t\) and the noisy source \(y_t\) diverge solely within regions enclosed by \(M_{t+1}\). To capture newly surfacing domain discrepancies between \(t+1\) and \(t\), the method composes a composite reference \(\tilde{y}_t = M_{t+1} y_t + (1 - M_{t+1}) x_t\) and evaluates the projection divergence using the model operator \(f_\theta\):

\[D_t(x_t, \tilde{y}_t) = |f_\theta(x_t, t) - f_\theta(\tilde{y}_t, t)|\]

(or \(1 - \text{SSIM}\) for perceptual structural divergence). Pixels where \(D_t(x_t, \tilde{y}_t) > \delta_t\) are identified as the newly emerging discrepancy \(\Delta M_t\). The active mask is updated via set union:

\[M_t = M_{t+1} \cup \Delta M_t\]

By accumulating discrepancies incrementally, the algorithm isolates the model's step-specific prediction errors while ignoring previously masked regions, allowing the mask to organically expand from coarse contours to fine textures without disturbing background pixels.

3. Multi-Batch Noise Averaging & Progressive Inpainting: Mitigating Noise Drift and Enforcing Boundary Harmony

Even within deterministic DDIM trajectories, random noise initialization can generate transient prediction outliers at high noise levels, producing noisy, fragmented mask borders. PM-Edit counteracts this by averaging the discrepancy maps \(D_t\) across \(N\) parallel image batches initialized with distinct random noise seeds prior to thresholding, yielding smooth and reliable mask contours.

Once the mask trajectory \([M_T, \dots, M_1]\) is established, naive one-pass spatial replacement produces seam artifacts along boundaries. To ensure seamless structural integration, PM-Edit incorporates a modified RePaint strategy: every 20 reverse steps, the process backtracks 10 steps with stochastic re-noising, repeated twice to facilitate bidirectional context exchange between generated and preserved content. Crucially, while standard RePaint enforces a static mask, PM-Edit applies the corresponding time-indexed mask \(M_t\) (dilated and blurred with kernel size 11 and Gaussian sigma 5.0) at each backward step. This grants wide structural freedom early in sampling while locking down fine boundary transitions near completion.

A Worked Example¶

Consider translating a wild animal image (Wild \(\to\) Dog) depicting a red fox: 1. Setup: The input image \(y\) is scaled to \(256 \times 256\). The target Dog diffusion model \(f_\theta\) and its precomputed discrepancy statistics \([\mu_{D_t}, \sigma_{D_t}]\) are loaded with \(\lambda = 1.2\). The initial mask \(M_T\) is initialized to zero everywhere. 2. Reverse Mask Evolution: Over 100 DDIM mask-generation steps, at early step \(t = 80\), severe visual divergence triggers \(\delta_{80}\) only across the fox's pointed snout and upright ear tips, generating a compact geometric mask \(\Delta M_{80}\). By intermediate step \(t = 50\), eye contours and cheek fur exhibit domain conflict, expanding \(M_{50}\) to encompass the facial region. By late step \(t = 20\), subtle whisker and muzzle differences are integrated into \(M_{20}\), while the natural forest background remains untouched (\(M = 0\)). 3. Progressive Inpainting: The accumulated masks are mapped across 1000 RePaint steps. During \(t \in [1000, 500]\), the masked fox snout is completely remodeled into a characteristic canine muzzle under strong dog-domain priors. During \(t \in [500, 0]\), repeated local backtracking and blurred mask edges blend dog fur texture seamlessly into the original background lighting, yielding a highly realistic dog portrait.

Key Experimental Results¶

Main Results¶

Evaluated on AFHQ (Wild \(\to\) Dog, Cat \(\to\) Dog) and Celeba-HQ (Male \(\to\) Female), PM-Edit was quantitatively benchmarked against prominent unsupervised I2I baselines. Among methods sharing the identical diffusion backbone (ILVR, SDEdit, EGSDE, CycleDiffusion), PM-Edit demonstrates consistent superiority, particularly on KID and LPIPS, which correlate more closely with human visual perception than FID and SSIM.

Task	Method	Source Trained	FID \(\downarrow\)	KID \(\times 10^3 \downarrow\)	SSIM \(\uparrow\)	LPIPS \(\downarrow\)
Wild \(\to\) Dog	CUT	Yes	92.94	48.0	0.592	-
	Santa	Yes	74.93	31.6	0.453	-
	UNSB	Yes	86.72	40.2	0.524	-
	SDDM	Yes	57.38	-	0.328	-
	ILVR	No	81.34 \(\pm\) 1.44	36.6 \(\pm\) 1.56	0.289 \(\pm\) 0.001	0.530 \(\pm\) 0.001
	SDEdit	No	69.38 \(\pm\) 1.10	27.6 \(\pm\) 1.10	0.343 \(\pm\) 0.001	0.515 \(\pm\) 0.001
	EGSDE	Yes	57.89 \(\pm\) 0.62	19.2 \(\pm\) 0.98	0.363 \(\pm\) 0.001	0.504 \(\pm\) 0.001
	CycleDiffusion	Yes	56.10 \(\pm\) 0.59	19.5 \(\pm\) 1.02	0.479 \(\pm\) 0.001	0.465 \(\pm\) 0.001
	PM-Edit (Ours)	No	55.64 \(\pm\) 0.61	17.2 \(\pm\) 0.92	0.479 \(\pm\) 0.001	0.445 \(\pm\) 0.001
Cat \(\to\) Dog	ILVR	No	75.38 \(\pm\) 1.49	31.5 \(\pm\) 1.95	0.359 \(\pm\) 0.001	0.507 \(\pm\) 0.001
	SDEdit	No	73.72 \(\pm\) 1.01	28.1 \(\pm\) 1.13	0.418 \(\pm\) 0.001	0.495 \(\pm\) 0.001
	EGSDE	Yes	63.37 \(\pm\) 0.71	21.4 \(\pm\) 1.21	0.437 \(\pm\) 0.001	0.471 \(\pm\) 0.001
	CycleDiffusion	Yes	58.87 \(\pm\) 0.69	20.4 \(\pm\) 1.10	0.557 \(\pm\) 0.001	0.426 \(\pm\) 0.001
	PM-Edit (Ours)	No	61.31 \(\pm\) 0.73	18.3 \(\pm\) 1.80	0.551 \(\pm\) 0.001	0.415 \(\pm\) 0.001
Male \(\to\) Female	ILVR	No	60.17 \(\pm\) 0.33	47.30 \(\pm\) 1.22	0.510 \(\pm\) 0.001	0.390 \(\pm\) 0.001
	SDEdit	No	54.91 \(\pm\) 0.47	47.70 \(\pm\) 1.34	0.577 \(\pm\) 0.001	0.361 \(\pm\) 0.001
	EGSDE	Yes	48.50 \(\pm\) 0.24	45.96 \(\pm\) 0.96	0.579 \(\pm\) 0.001	0.357 \(\pm\) 0.001
	CycleDiffusion	Yes	45.04 \(\pm\) 0.23	45.11 \(\pm\) 0.98	0.564 \(\pm\) 0.001	0.369 \(\pm\) 0.001
	PM-Edit (Ours)	No	50.99 \(\pm\) 0.25	42.10 \(\pm\) 0.91	0.586 \(\pm\) 0.001	0.356 \(\pm\) 0.001

Ablation Study¶

Systematic ablations conducted on Wild \(\to\) Dog isolate the necessity of each architectural component (left) and assess sensitivity to the threshold coefficient \(\lambda\) (right).

Component Ablation Setting	KID \(\times 10^3 \downarrow\)	LPIPS \(\downarrow\)	Note	Threshold \(\lambda\)	FID \(\downarrow\)	KID \(\times 10^3 \downarrow\)	SSIM \(\uparrow\)	LPIPS \(\downarrow\)
Default PM-Edit (Full)	17.2	0.445	Dynamic mask + adaptive statistical threshold	1.0	54.12	17.1	0.453	0.459
Constant mask \(M_{0.5T}\)	33.3	0.309	Static mask computed at midpoint \(t=0.5T\)	1.2 (default)	55.64	17.2	0.479	0.445
Constant mask \(M_0\)	13.1	0.533	Static mask using terminal boundary throughout	1.4	58.34	19.4	0.497	0.442
Constant threshold (constant \(\delta_t\))	10.5	0.698	Fixed threshold equivalent to \(t=0.5T\) throughout	1.6	61.35	21.4	0.521	0.413
Single-batch noise (\(N=1\))	16.2	0.617	Mask evaluated without multi-seed averaging	1.8	62.89	22.6	0.531	0.406
Without RePaint backtracking	28.0	0.437	Standard single-pass latent inpainting	-	-	-	-	-

Key Findings¶

Crucial Role of Time-Varying Masks: Emulating DiffEdit via a static midpoint mask (\(M_{0.5T}\)) degrades KID to 33.3 due to insufficient generative freedom in the early stages. Enforcing the terminal mask (\(M_0\)) across the entire process lowers KID to 13.1 but causes LPIPS to deteriorate sharply to 0.533 as background and invariant structures are needlessly overwritten. Progressive masking is essential to balance both objectives.
Dynamic Statistical Thresholding: Fixing the threshold \(\delta_t\) across all noise levels causes large discrepancies in early noisy steps to mark the entire image as anomalous, sending LPIPS to 0.698 and destroying source image faithfulness.
Tunable Realism-Faithfulness Pareto Frontier: The threshold parameter \(\lambda\) provides fine-grained control over the translation trade-off. Lower values (\(\lambda = 1.0\)) relax the anomaly threshold, yielding larger masked areas, enhanced generation freedom, and superior realism scores (FID 54.12, KID 17.1). Higher values (\(\lambda = 1.8\)) enforce conservative masking, retaining more source structures and boosting faithfulness (SSIM 0.531, LPIPS 0.406).

Highlights & Insights¶

Source-Agnostic Elegance: While prior unsupervised I2I frameworks depend on heavy joint training or dual-domain inversion schedules, PM-Edit proves that a single target-domain generative prior provides sufficient signal to localize cross-domain discrepancies via reverse projection prediction divergence.
Empirical Statistical Standardization: Instead of manually tuning arbitrary threshold heuristics across hundreds of time steps, calibrating against the target domain's empirical prediction variance (\(\mu + \lambda \sigma\)) creates an automated, robust baseline across arbitrary noise levels.
Latent-Space Generalizability: The framework transfers seamlessly to Latent Diffusion Models (Stable Diffusion v1.5 on PIE-Bench), where operating in VAE latent space eliminates the need for RePaint backtracking and dramatically reduces execution overhead.

Limitations & Future Work¶

Shared Modality Assumption: The current framework assumes that the source and target domains share comparable spatial layouts and modalities (e.g., animal faces or human portraits). Under severe topological changes or cross-modal domain shifts (e.g., sketch-to-photo or zebra-to-car), raw pixel/latent discrepancy metrics can collapse.
Inference Latency: Generating high-fidelity translations in pixel space entails 100 DDIM mask-generation steps with \(N=10\) parallel noise seeds, followed by 1000 RePaint inpainting steps, incurring substantially higher compute costs than feed-forward GAN translators.
Future Directions: Adapting latent-aware adaptive masking to few-step diffusion generators (such as Consistency Models or Flow Matching) and investigating feature-space discrepancy metrics with higher semantic invariance represent promising research frontiers.

vs SDEdit / ILVR: SDEdit and ILVR are also source-agnostic but lack spatial adaptivity, either corrupting fine details via global noise injection or over-constraining geometric transformations via low-frequency filtering. PM-Edit achieves spatially selective modification via dynamic minimal masking.
vs DiffEdit: DiffEdit relies on a single time step and a fixed threshold to derive a static mask. PM-Edit introduces time-dependent statistical thresholding and iterative union accumulation, allowing masks to adapt dynamically to evolving noise scales.
vs EGSDE / CycleDiffusion: EGSDE and CycleDiffusion require pretraining on both source and target domains, preventing zero-shot application to new source distributions. PM-Edit operates entirely without source-domain supervision.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [A training-free, source-agnostic masking paradigm grounded in target-domain discrepancy statistics across reverse diffusion]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across AFHQ, Celeba-HQ, and PIE-Bench benchmarks with exhaustive metric coverage and ablation studies]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, transparent rationale for statistical thresholding, and well-structured empirical validation]
Value: ⭐⭐⭐⭐☆ [Provides a highly practical, plug-and-play solution for unsupervised image editing that generalizes readily to modern latent diffusion backbones]