Accelerated Likelihood Maximization for Diffusion-based Versatile Content Generation¶

Conference: ECCV 2026
arXiv: 2606.31323
Code: http://hleephilip.github.io/ALM (Project website, code to be released)
Area: Diffusion Models / Image Generation
Keywords: Diffusion Models, Likelihood Maximization, Training-Free, Content Completion, Accelerated Sampling

TL;DR¶

ALM proposes a completely training-free sampling strategy for diffusion models. It performs explicit likelihood maximization optimization on unobserved regions during the reverse diffusion process, and leverages the property that "updates of adjacent iterations are approximately equal" to collapse \(N\) iterations into a single update. It significantly outperforms prior state-of-the-art methods at zero training cost across multi-modal tasks including image inpainting/outpainting, human motion completion, 3D texture generation, and long video generation.

Background & Motivation¶

Diffusion models have demonstrated remarkable capabilities in standard generative tasks such as unconditional or text-to-image/text-to-video generation. However, practical applications often require generating complete content starting from partially given inputs—image inpainting needs to fill missing parts based on known regions, panorama generation requires outpainting from an initial patch, human motion completion needs to infer full motion sequences from partial keyframes, and 3D texture generation needs to maintain texture consistency across multiple views. These generation tasks based on partial observations are collectively referred to as versatile content generation. The most straightforward approach is to retrain or fine-tune models for each task. But this not only incurs high computational costs and requires massive labeled data, but also generalizes poorly—an inpainting model trained on a specific dataset can run into severe issues when transferred to other modalities.

Training-free diffusion synchronization methods (SyncTweedies, SyncSDE, StochSync) attempt to avoid retraining by coupling multiple diffusion trajectories to maintain global consistency. However, their core limitation is that they only impose constraints on the generated known regions and perform no explicit optimization on the missing regions, defaulting to the assumption that the diffusion process will automatically yield reasonable results. This assumption may barely hold when the missing region is small, but fails completely once the missing area becomes large (e.g., image outpainting, long video stitching), leading to semantic contradictions, color flickering, and global inconsistency artifacts. While schemes like SyncSDE provide a probabilistic framework to explain their guidance mechanism, their guidance term only acts on the known regions \((1-M)\odot Y_t\) and exerts no control over the missing regions \(M\odot Y_t\).

The entry point of this paper is direct: since the issue lies in the unobserved regions being left "unsupervised", they should be directly optimized with explicit constraints rather than relying solely on indirect guidance. Core Idea: Reframe diffusion synchronization as explicit likelihood maximization of unobserved variables. In each reverse diffusion step, a closed-form gradient update is derived via a dual objective of conditional likelihood (local consistency) and joint log-density (global harmony) to directly optimize the missing regions. Furthermore, leveraging Lipschitz continuity, the authors prove that updates of adjacent iterations are approximately equal, thereby collapsing \(N\) iterations into a single-step calculation and achieving approximately a 185x speedup.

Method¶

Overall Architecture¶

ALM is fully embedded within the reverse diffusion process of pretrained diffusion models, preserving the network architecture without introducing extra training parameters. Given the pre-generated content \(X\) (known region, such as unmasked parts of an image) and a binary mask \(M\) (where 1 indicates unobserved regions), ALM performs two core steps at each diffusion timestep \(t\): first, it applies a likelihood maximization update to the unobserved variables \(Y_t\), and then executes an improved DDIM reverse process to sample \(Y_{t-1}\). The figure below details the complete workflow within a single timestep:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Timestep t<br/>Known content X_t<br/>Unobserved variable Y_t<br/>Mask M"] --> B["Construct mixed variable<br/>E_t = X_t⊙(1-M) + Y_t⊙M"]
    B --> C["Dual-path noise prediction<br/>ε_θ(Y_t) and ε_θ(E_t)"]
    C --> D["ALM Likelihood Maximization<br/>ΔY_t = M⊙(w₁(ε_θ(Y_t)-ε_θ(E_t))<br/>- w₂ε_θ(E_t))"]
    D --> E["Update unobserved region<br/>Y_t ← Y_t + M⊙ΔY_t"]
    E --> F["Improved DDIM reverse process<br/>with known region guidance<br/>Y_{t-1} ← DDIM(Y_t) + w₁(1-M)⊙(X_t-Y_t)"]
    F --> G["Sample Y_{t-1}"]
    G -->|Loop to t=0| A

The core lies in ALM simultaneously optimizing two complementary objectives in each update: the conditional likelihood term aligns the missing content with the known context, while the joint log-density term ensures the overall mixed result (known + generated) resides in the high-density region of the data distribution, remaining realistic and harmonious. The relative weights of the two objectives are controlled by hyperparameters \(w_1\) and \(w_2\).

Key Designs¶

1. Explicit Likelihood Maximization of Unobserved Regions: From Indirect Guidance to Direct Optimization

Prior diffusion synchronization methods solely applied guidance mechanisms to known regions—they constrained the denoising trajectory of known regions to indirectly affect missing regions without imposing any direct constraints on \(M \odot Y_t\). In SyncSDE, for instance, the conditional term in the score decomposition \(\nabla \log p(Y_t|X_t) = \nabla \log p(Y_t) + \nabla \log p(X_t|Y_t)\) only depends on known regions, leaving missing regions to be completed automatically by the diffusion model's prior. This assumption barely holds for small missing regions but produces obvious global inconsistencies when the missing area is large.

ALM introduces an extra update step \(\Delta Y_t\) for the unobserved variables \(Y_t\) at each timestep \(t\). This update is derived by minimizing a composite likelihood objective: the first term \(-\lambda_1 \log p(X_t, M | Y_t)\) encourages the missing content to align with the known context condition, while the second term \(-\lambda_2 \log p(X_t, M, Y_t)\) pushes the entire composition of known and unknown regions to lie in high-density regions of the data distribution (i.e., looking realistic overall). Under a Taylor expansion and score-based substitution, a closed-form solution is derived:

\[\Delta \mathbf{Y}_t = \mathbf{M} \odot \big( w_1 (\epsilon_\theta(\mathbf{Y}_t) - \epsilon_\theta(\mathbf{E}_t)) - w_2 \epsilon_\theta(\mathbf{E}_t) \big)\]

where \(E_t = X_t \odot (1-M) + Y_t \odot M\) is the mixed variable. This formula consists of two intuitive components: \(\epsilon_\theta(Y_t) - \epsilon_\theta(E_t)\) measures the difference in noise prediction between "viewing only the missing region vs. viewing the global context," allowing the missing region to perceive global information; \(\epsilon_\theta(E_t)\) drives the mixed result toward the data manifold. Designing \(\lambda_1 > \lambda_2\) ensures local consistency takes precedence over global harmony—aligning the context first, then pursuing overall realism. Ablation studies confirm that removing either of the terms decreases performance, and their relative contributions vary across different backbones.

2. One-Step Acceleration Strategy: Collapsing N Iterations into a Single Update

The aforementioned likelihood maximization update theoretically requires multiple iterations: in each iteration, calculating \(\Delta Y_t^i\) and updating \(Y_t^i = Y_t^{i-1} + M \odot \Delta Y_t^{i-1}\), repeating \(N\) times before moving to the next DDIM step. This introduces \(O(N)\) computational overhead, which is prohibitive for reverse diffusion sampling that typically requires 50-1000 steps.

A key insight of ALM is that when the hyperparameters \(\lambda_1, \lambda_2\) are chosen small enough to satisfy the \(\ell_2\) norm condition of the Taylor expansion \(||\Delta Y_t^i|| \ll 1\), then because the noise prediction network \(\epsilon_\theta\) is \(L\)-Lipschitz continuous, the updates of two consecutive iterations satisfy \(||\Delta Y_t^{i+1} - \Delta Y_t^i|| \le L(2\lambda_1 + \lambda_2)||\Delta Y_t^i|| = O(||\Delta Y_t^i||)\). This indicates adjacent updates are approximately equal. Therefore, the total effect of \(N\) iterations can be directly approximated by \(N\) times the first-step update:

\[\Delta Y_t \approx N \cdot \Delta Y_t^1 = M \odot (w_1'(\epsilon_\theta(Y_t) - \epsilon_\theta(E_t)) - w_2\epsilon_\theta(E_t))\]

where \(w_1' = N \cdot \lambda_1\) and \(w_2 = N \cdot \lambda_2\) serve as directly tunable hyperparameters (in practice, \(\lambda_1 = 2 \times 10^{-3}, \lambda_2 = 10^{-5}\) correspond to \(w_1 = 1.0, w_2 = 0.005\)). This approximation collapses \(N\) iterations into a single step, achieving approximately a 185x runtime speedup in experiments, while the quantitative metrics (LPIPS, MS-SSIM, FSIM) remain almost identical. Interestingly, ablation studies show that the automated single-step version yields slightly better metrics than the iterative version (MS-SSIM 0.351 vs. 0.298), indicating that the single-step approximation finds better optimization directions under limited steps.

Loss & Training¶

ALM requires no training whatsoever, only needing to tune the two hyperparameters \(w_1\) and \(w_2\) during DDIM sampling. Both decay over timestep \(t\) proportional to \(\sigma_t = \sqrt{(1-\alpha_{t-1})/(1-\alpha_t)} \cdot \sqrt{1 - \alpha_t/\alpha_{t-1}}\) to ensure the small-update assumption still holds in the second half of reverse diffusion (when samples are close to clean data). Typical config: \(w_1 = 1.0, w_2 = 0.005\) for image inpainting; for flow matching frameworks like FLUX, the noise prediction \(\epsilon_\theta\) is replaced by velocity field prediction \(v_θ\), and a decay strategy of \(\sigma_t = t^4\) is adopted. The method is robust to hyperparameters (performance remains stable within the range of \(w_1 \in [0.5, 1.5], w_2 \in [0.001, 0.01]\)).

Key Experimental Results¶

Main Results¶

Image Inpainting (Stable Diffusion Backbone, 50 DDIM steps)

Dataset	Method	LPIPS↓	MSE↓	M-SSIM↑	MS-SSIM↑	FSIM↑
AFHQ	SyncSDE	0.304	0.172	0.302	0.641	0.778
AFHQ	BrushNet (Requires Training)	0.316	0.216	0.256	0.589	0.741
AFHQ	SD Inpainting (Requires Training)	0.292	0.140	0.295	0.623	0.757
AFHQ	ALM	0.283	0.143	0.351	0.689	0.796
CelebA-HQ	SyncSDE	0.292	0.159	0.341	0.661	0.781
CelebA-HQ	BrushNet (Requires Training)	0.274	0.195	0.347	0.638	0.759
CelebA-HQ	SD Inpainting (Requires Training)	0.268	0.130	0.368	0.659	0.763
CelebA-HQ	ALM	0.251	0.126	0.417	0.732	0.813

At zero training cost, ALM comprehensively outperforms both training-required models (BrushNet/PowerPaint/SDI) and training-free models (SyncSDE/HD-Painter). Of particular note is the substantial lead in M-SSIM (which evaluates generation quality exclusively in the unobserved regions)—directly highlighting the effectiveness of explicitly optimizing the missing regions.

Image Outpainting (Outpainting → 2048×512)

Method	FID↓	KID↓(×10³)	Aesthetic Score↑	Q-Align↑
SyncTweedies	85.95	58.36	6.104	4.550
SyncSDE	85.82	51.84	6.127	4.542
StochSync	113.21	92.10	6.026	4.546
ALM	83.41	42.98	6.133	4.581

Ablation Study¶

Configuration	AFHQ M-SSIM↑	AFHQ FSIM↑	AFHQ LPIPS↓	Description
w/o ALM (Pure SyncSDE)	0.300	0.782	0.295	No unobserved region optimization
w/o Conditional Likelihood Term	0.323	0.787	0.295	Removes local consistency constraint
w/o Joint Log-Density Term	0.327	0.793	0.284	Removes global harmony constraint
w/o Acceleration (N=500 iterations)	0.298	0.781	0.298	Iterative update without acceleration
Full ALM	0.351	0.796	0.283	Dual objectives + One-step acceleration

The ablation of the acceleration strategy is particularly intriguing: removing acceleration (running 500 iterations) actually yields slightly worse metrics than the accelerated version (M-SSIM 0.298 vs. 0.351), yet the difference in runtime is staggering—9.9 seconds for the accelerated version versus 1854 seconds for the non-accelerated version (approx. 185x speedup). The authors hypothesize that because the Lipschitz bound is conservative in practice, the single-step approximation instead finds a better optimization direction within a finite number of steps.

Key Findings¶

The two objective terms contribute differently across backbones: for conditional diffusion models (Stable Diffusion), the conditional likelihood term is more crucial, whereas for unconditional diffusion models (RePaint), the joint log-density term dominates. The dual-objective design allows ALM to automatically adapt to different backbone architectures.
ALM achieves new SOTA across all five tasks (image inpainting with 4 datasets, image outpainting, human motion completion with 3 scenarios, 3D texture generation, and long video generation), demonstrating the generality of the method.
Integrating with FLUX (a flow matching framework) requires adjusting the decay strategy (\(\sigma_t^4\)), reaching an MS-SSIM of 0.277 on AFHQ; while slightly lower than SD/SDXL backbones, it still significantly outperforms unoptimized FLUX-Inpainting.
In 3D texture generation, ALM registers an FID of 155.46 and a KID of \(74.41 \times 10^{-3}\), outperforming not only all synchronization methods but also task-specific approaches like Paint-it, TexPainter, and TEXTure, proving the competitiveness of training-free methods in the 3D domain.

Highlights & Insights¶

The blind spot of "no control over the missing regions" is accurately identified: Prior diffusion synchronization methods formulated guidance mechanisms solely on known regions. ALM is the first to explicitly recognize that "the root issue lies in the unknown regions, so optimization should target them directly." This paradigm shift from indirect guidance to direct optimization is the fundamental reason for its substantial lead.
The dual-objective design cleverly leverages the complementarity of probabilistic modeling: Conditional likelihood models local alignment while joint log-density models global harmony. These two seemingly conflicting objectives are unified under a composite likelihood framework with prioritized weights (\(\lambda_1 > \lambda_2\)). This hierarchical optimization approach (local first, then global) can be generalized to other conditional generation tasks (such as video inpainting or point cloud completion).
The mathematical derivation of the acceleration strategy is highly impressive: Proving that the updates of adjacent iterations are approximately equal using Lipschitz continuity, thereby collapsing them into a single step, is a rigorous mathematical derivation rather than just an empirical finding. As shown in Figure 18 of Appendix D, the discrepancy between adjacent updates remains consistently minimal regardless of mask shape, validating the reliability of this derivation.
Exceptional cross-modal generalization with compelling experimental design: Evaluated extensively across image (SD/SDXL/FLUX), human motion (U-Net diffusion), 3D (ControlNet), and video (LaVie) modalities, requiring only adjustments to the hyperparameter decay strategy per modality. This demonstrates the immense potential of training-free methods in versatile content generation.

Limitations & Future Work¶

Although the acceleration strategy is highly effective, ALM still requires two forward passes (to predict \(\epsilon_\theta(Y_t)\) and \(\epsilon_\theta(E_t)\) respectively) in each reverse diffusion step, meaning the single-step compute is twice that of standard sampling. This may still be a bottleneck in ultra-low latency scenarios.
For flow matching frameworks (like FLUX), adapting ALM requires additional mathematical formulation translations and customized hyperparameter schedules (\(\sigma_t^4\)), falling short of being completely plug-and-play. Formulating a more unified scheduling strategy represents a direction for future improvement.
The core assumption of the acceleration strategy relies on Lipschitz continuity. However, the Lipschitz constants of practical neural networks are difficult to constrain tightly, rendering the theoretical upper bounds somewhat loose. Although validated empirically, a more rigorous analysis would enhance credibility.
The hyperparameters \((w_1, w_2)\) still require manual tuning across different tasks, lacking auto-recommendation mechanisms. Learning the statistical properties of noise prediction discrepancies across tasks could be a direction for automated hyperparameter tuning.

vs SyncSDE: Both belong to the diffusion synchronization paradigm, but the conditional term \(\nabla \log p(X_t|Y_t)\) in SyncSDE's score decomposition only acts on known regions while leaving missing regions unconstrained. ALM addresses this with the explicit likelihood maximization of missing regions, fundamentally overcoming SyncSDE's core limitation. The \((1-M)\odot(X_t - Y_t)\) guidance term in Eq. (2) of SyncSDE is preserved in ALM, while the additionally added \(\Delta Y_t\) is ALM's key innovation.
vs SyncTweedies: Based on an empirical search of 60 strategies, it lacks underlying mathematical explanation. ALM derives a closed-form solution from likelihood maximization, offering a clearer mechanism and significantly superior performance.
vs HD-Painter / Reconstruction Guidance: These training-free methods are tailored for specific tasks (attention weightings / L2 reconstruction losses) and cannot generalize to other modalities. ALM offers a unified optimization framework with broader applicability.
vs BrushNet / PowerPaint / CondMDI: These training-required methods achieve good results on their respective tasks, but at the cost of requiring massive data and computation resources for each task. ALM outperforms them at zero training cost, demonstrating the competitiveness and potential of the training-free paradigm.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Redefining diffusion synchronization as an explicit likelihood maximization of missing regions is fundamentally different from prior work in paradigm. The mathematical derivation of the acceleration strategy is concise, elegant, and empirically validated.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 5 tasks \(\times\) multiple backbones (SD, SDXL, FLUX, Unconditional, ControlNet, LaVie) \(\times\) multiple datasets, accompanied by extensive ablation and sensitivity analyses, as well as score estimation mask perturbation verification in Appendix C.
Writing Quality: ⭐⭐⭐⭐ Follows a clear logical flow from motivation to derivation and evaluation. However, the derivations in Section 3.3 might feel slightly abrupt for readers unfamiliar with score matching. The verification experiments in the appendix are highly recommended.
Value: ⭐⭐⭐⭐⭐ In modern applications of diffusion models, "completion of content from partial inputs" is a ubiquitous need. ALM's training-free solution can be directly integrated into existing pipelines, offering extremely high practical value.