Skip to content

Learning to Corrupt for Better Restoration

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Restoration
Keywords: Image Restoration, Diffusion Model, Super-Resolution, Knowledge Distillation, Input-Aware Corruption

TL;DR

To eliminate heuristic corruption strategies that diverge from pretrained diffusion trajectories in one-step image restoration, IAC-IR predicts sample-specific optimal timesteps and noise via intrinsic Gaussian statistical supervision and denoiser consistency, seamlessly integrating them into score distillation to achieve superior one-step perceptual reconstruction.

Background & Motivation

Image restoration (IR) aims to recover photorealistic high-quality (HQ) images from degraded low-quality (LQ) inputs, serving as an indispensable foundation for computational photography and perceptual enhancement. Early approaches centered on Generative Adversarial Networks (GANs) mitigated the over-smoothing artifacts typical of pixel-wise regression, yet they suffered from training instability and severe mode collapse. Pretrained diffusion models have recently emerged as the dominant generative backbone for image restoration, excelling in synthesizing sharp, high-frequency textural details. Early diffusion-based methods conditioned the generative process on LQ images while executing a standard multi-step reverse denoising schedule initialized from pure Gaussian noise. This paradigm not only incurs prohibitive sampling latency but also discards and redundantly regenerates informative low-frequency structures already preserved in the LQ input.

To accelerate inference, recent efforts have shifted toward one-step diffusion restoration models. However, pretrained diffusion models are fundamentally grounded in the mathematical assumption of progressively recovering clean data distributions from Gaussian noise-corrupted trajectories. Existing one-step methods routinely resort to oversimplified, heuristic corruption strategies: directly feeding the uncorrupted LQ latent into the denoiser, injecting randomly sampled Gaussian noise, or imposing an arbitrary fixed timestep. Such rigid strategies overlook the heterogeneous, spatially varying nature of real-world degradations, causing the corrupted input to deviate severely from the pretrained diffusion manifold. Furthermore, in conventional score-based distillation pipelines, target diffusion scores calculated from randomly corrupted samples introduce excessive gradient variance, yielding unstable distillation signals and suboptimal fine-grained details.

The core tension lies in the fact that real degraded images lack ground-truth corruption factors (neither an optimal timestep nor a reference noise map exists), precluding direct supervised regression; without proper input-aligned corruption, the frozen generative prior cannot be effectively activated in a single forward pass. The core idea is to leverage the intrinsic Gaussian statistics and denoising consistency of pretrained diffusion models to construct an indirect self-supervision framework that predicts input-specific timesteps \(\hat{t}\) and noise \(\hat{\epsilon}\), directly incorporating these deterministic corruption factors into score-based distillation for stable, high-fidelity one-step restoration.

Method

Overall Architecture

The entire IAC-IR framework operates in the latent space of a pretrained VAE. A degraded observation \(I_{\text{LQ}}\) is first mapped into an optimally corrupted latent representation \(\mathbf{X}\) via an image encoder \(E_\theta\). Next, a lightweight timestep estimator \(\mathcal{T}\) predicts the input-specific timestep \(\hat{t}\), while a LoRA-adapted UNet denoiser \(\mathcal{U}_\phi\) decomposes \(\mathbf{X}\) into the clean latent \(\hat{\mathbf{x}}_0\) and the noise component \(\hat{\epsilon}\) conditioned on \(\hat{t}\). To overcome the absence of explicit corruption annotations during training, an analytical candidate residual search evaluates Gaussian statistics against the clean target to yield a pseudo-ground-truth timestep \(t_{\text{GT}}\), supervised via cross-entropy. Concurrently, a frozen pretrained denoiser \(\mathcal{U}_\psi\) enforces self-supervised noise consistency. Finally, the estimated clean latent is decoded into the restored image \(I_{\text{IR}}\), optimized jointly through an input-aware score distillation objective alongside standard perceptual and adversarial criteria.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Degraded Input $I_{\text{LQ}}$"] --> B["Latent Encoder $E_\theta$<br/>Map to Corrupted Latent $\mathbf{X}$"]
    B --> C["Gaussian Statistics Search for Optimal $t_{\text{GT}}$"]
    B --> D["Timestep Estimator $\mathcal{T}$<br/>Predict Input-Aware Timestep $\hat{t}$"]
    C -->|Cross-Entropy Loss $\mathcal{L}_t$| D
    B & D --> E["Learnable Denoiser $\mathcal{U}_\phi$<br/>Predict Noise $\hat{\epsilon}$ & Clean Latent $\hat{\mathbf{x}}_0$"]
    E --> F["Frozen Denoiser Consistency<br/>Optimize $\mathcal{L}_\epsilon$ on Corrupted Latent $\tilde{\mathbf{x}}_{\hat{t}}$"]
    E --> G["Input-Aware Score Distillation $\mathcal{L}_{\text{distill}}$<br/>Guide Encoder Update via $(\hat{t}, \hat{\epsilon})$"]
    E --> H["Decoder & Composite Objectives<br/>Reconstruct Image via $\mathcal{L}_1 + \mathcal{L}_{\text{VGG}} + \mathcal{L}_{\text{adv}}$"]

Key Designs

1. Gaussian Statistics Search for Optimal \(t_{\text{GT}}\): Indirect pseudo-ground-truth discovery

To ensure that the encoded latent \(\mathbf{X}\) lands precisely on the pretrained forward diffusion trajectory, the representation should adhere to the linear mixing formulation \(\mathbf{X} = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\epsilon\). Because no explicit timestep label exists for a given degraded image, the authors capitalize on the foundational property that pretrained diffusion models are trained on true standard Gaussian noise \(\mathcal{N}(0, \mathbf{I})\). For each candidate timestep \(t \in \{0, \dots, T-1\}\), the candidate residual noise component is algebraically deduced from the clean latent \(\mathbf{x}_0\):

\[\mathbf{r}_t = \frac{\mathbf{X} - \sqrt{\bar{\alpha}_t}\mathbf{x}_0}{\sqrt{1-\bar{\alpha}_t}}\]

The deviation of \(\mathbf{r}_t\) from an ideal normal distribution is evaluated via a statistical metric \(\mathcal{G}(\mathbf{r}_t)\) combining sample mean \(\mu\), variance \(\sigma^2\), skewness \(\gamma\), and kurtosis \(\kappa\):

\[\mathcal{G}(\mathbf{r}_t) = \|\mu(\mathbf{r}_t)\|^2 + \|\sigma^2(\mathbf{r}_t) - 1\|^2 + \|\gamma(\mathbf{r}_t)\|^2 + \|\kappa(\mathbf{r}_t) - 3\|^2\]

The optimal pseudo-ground-truth timestep is selected as the candidate minimizing this Gaussianity penalty: \(t_{\text{GT}} = \arg\min_t \mathcal{G}(\mathbf{r}_t)\). This objective mechanism circumvents heuristic manual step tuning and provides a well-posed supervision signal for training.

2. Timestep Estimator \(\mathcal{T}\) & Cross-Entropy Supervision: Inference-time adaptive prediction

During deployment, the clean reference \(\mathbf{x}_0\) is unavailable, making direct residual search impossible. To infer the optimal corruption level from the degraded input alone, IAC-IR introduces a lightweight Vision Transformer module \(\mathcal{T}\) (comprising 12 ViT layers, amounting to roughly 2% of the overall model parameters). The estimator processes a sequence formed by concatenating a learnable class token with the flattened spatial tokens of latent \(\mathbf{X}\), followed by a linear classification head generating logits over all \(T\) discrete timesteps. The predicted timestep corresponds to the channel of maximum activation, \(\hat{t} = \arg\max_t \mathcal{T}(\mathbf{X})_t\). During training, this prediction is directly supervised by \(t_{\text{GT}}\) via standard cross-entropy:

\[\mathcal{L}_t = \text{CE}(\mathcal{T}(\mathbf{X}), t_{\text{GT}})\]

This design empowers the model to deterministically identify the optimal signal-to-noise ratio in a single inference forward pass.

3. Frozen Denoiser Consistency: Self-supervised content-preserving noise learning

Beyond the scalar timestep magnitude, the spatial structure of injected noise governs whether the reverse diffusion trajectory can reconstruct recoverable input features. While prior works like InvSR trained noise implicitly without explicit guidance, such unconstrained setups frequently cause the estimated noise to deviate from Gaussian properties. IAC-IR introduces an explicit self-supervised consistency loop using a frozen pretrained UNet \(\mathcal{U}_\psi\). Specifically, the learnable LoRA-equipped denoiser \(\mathcal{U}_\phi\) estimates the noise \(\hat{\epsilon}\) and clean latent \(\hat{\mathbf{x}}_0\) from \((\mathbf{X}, \hat{t})\). Next, a synthetic noisy latent \(\tilde{\mathbf{x}}_{\hat{t}} = \sqrt{\bar{\alpha}_{\hat{t}}}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_{\hat{t}}}\hat{\epsilon}\) is formed by corrupting the clean latent \(\mathbf{x}_0\) using the predicted \((\hat{t}, \hat{\epsilon})\). Passing \(\tilde{\mathbf{x}}_{\hat{t}}\) through the frozen denoiser \(\mathcal{U}_\psi\) produces a reconstructed noise target \(\tilde{\epsilon} = \mathcal{U}_\psi(\tilde{\mathbf{x}}_{\hat{t}}, \hat{t})\). The network is then optimized with a consistency loss:

\[\mathcal{L}_\epsilon = \|\tilde{\epsilon} - \hat{\epsilon}\|^2\]

This constraint enforces that \(\hat{\epsilon}\) mirrors the exact noise characteristics that the frozen diffusion backbone is natively trained to denoise, simultaneously preventing distributional drift and preserving input semantic layout.

4. Input-Aware Score Distillation: Deterministic corruption for low-variance gradients

Standard score-based diffusion distillation computes target scores on latents corrupted with randomly sampled timesteps and noise, which introduces substantial stochasticity and erratic gradient updates in single-step settings. IAC-IR repurposed the learned input-adaptive parameters directly within the distillation objective. The encoded latent \(\mathbf{X}\) naturally acts as the deterministic corrupted input, yielding a student clean estimate \(\hat{\mathbf{x}}_0 = (\mathbf{X} - \sqrt{1-\bar{\alpha}_{\hat{t}}}\mathcal{U}_\phi(\mathbf{X}, \hat{t})) / \sqrt{\bar{\alpha}_{\hat{t}}}\). Concurrently, passing the identical latent \(\mathbf{X}\) and predicted timestep \(\hat{t}\) through the frozen teacher denoiser \(\mathcal{U}_\psi\) yields the target clean estimate \(\hat{\mathbf{x}}_0^\psi\). The score distillation loss is defined as:

\[\mathcal{L}_{\text{distill}}(\theta) = \mathbb{E}\left[\|\hat{\mathbf{x}}_0 - \hat{\mathbf{x}}_0^\psi\|^2\right]\]

This objective exclusively updates the parameters of encoder \(E_\theta\). Because the distillation targets are calculated along a deterministic, input-aligned trajectory, gradient variance is sharply curtailed, facilitating stable optimization and the synthesis of crisp high-frequency textures.

Loss & Training

The global objective integrates four specialized loss formulations: 1. Corruption Supervision: Cross-entropy timestep loss \(\mathcal{L}_t\) and noise consistency loss \(\mathcal{L}_\epsilon\); 2. Knowledge Distillation: The input-aware score distillation loss \(\mathcal{L}_{\text{distill}}\) updating encoder \(E_\theta\); 3. Fidelity & Perceptual Objectives: Pixel-wise \(\mathcal{L}_1\) loss and VGG-based perceptual loss \(\mathcal{L}_{\text{VGG}}\) between decoded output \(I_{\text{IR}}\) and ground-truth \(I_{\text{HQ}}\); 4. Adversarial Regularization: Patch-based adversarial loss \(\mathcal{L}_{\text{adv}}\) to align restored patches with natural image distributions.

Training utilizes Stable Diffusion 2.1 as the foundational backbone, optimizing with AdamW across four H100 GPUs for 300k iterations with a batch size of 4 per GPU. The initial learning rate is \(1\times 10^{-4}\), decayed by 0.5 every 100k iterations on \(512\times 512\) image patches.

Key Experimental Results

Main Results

IAC-IR was benchmarked against leading one-step diffusion restoration and super-resolution models on synthetic ImageNet degradation alongside various challenging real-world benchmarks (RealSR, DRealSR, RealLR200, RealPhoto, RealSet80). Representative quantitative results from Table 1 of the main paper are summarized below:

Dataset Metric IAC-IR (Ours) HYPIR InvSR OSEDiff Relative Advantage
RealSR PSNR (dB) ↑ 22.509 21.408 22.559 23.589 Competitive pixel fidelity
SSIM ↑ 0.672 0.656 0.685 0.707 Solid structural preservation
LIQE ↑ 4.668 3.970 4.039 4.068 Substantial gain over prior SOTA (+0.600)
MUSIQ ↑ 69.853 66.247 68.537 69.091 Top-ranked perceptual score
CLIPIQA ↑ 0.709 0.638 0.679 0.668 Highest natural visual alignment
MANIQA ↑ 0.589 0.476 0.455 0.472 Remarkable texture realism (+0.113)
ImageNet PSNR (dB) ↑ 22.187 21.116 22.018 23.056 Surpasses HYPIR and comparable to InvSR
LIQE ↑ 4.721 4.606 4.560 4.561 Best perceptual quality on synthetic benchmark
MUSIQ ↑ 73.546 72.693 72.382 71.751 Outperforms all competing one-step methods
MANIQA ↑ 0.617 0.561 0.469 0.459 Cleanest and sharpest details
RealPhoto LIQE ↑ 4.829 4.686 1.863 4.625 Robust performance on unscaled real images
MANIQA ↑ 0.635 0.581 0.298 0.483 Prevents severe degradation collapse seen in InvSR

Ablation Study

Ablation investigations systematically probed the impact of timestep prediction, noise supervision formulation, and distillation sampling strategies (reported on the RealSR benchmark from Tables 2, 3, and 5 of the original paper):

Table 1: Ablation on Timestep Selection Strategies (Paper Table 2)

Timestep Setting PSNR (dB) ↑ SSIM ↑ LIQE ↑ MUSIQ ↑ CLIPIQA ↑ MANIQA ↑ Mechanism Implication
Fixed \(t = 100\) 23.064 0.660 3.816 66.122 0.694 0.559 High fidelity but insufficient generative restoration
Fixed \(t = 200\) 22.077 0.669 4.448 69.756 0.684 0.559 Moderate compromise
Fixed \(t = 400\) 20.492 0.615 4.571 69.927 0.709 0.588 Enhanced generation at cost of severe distortion
Adaptive \(\hat{t}\) (Ours) 22.509 0.672 4.668 69.853 0.709 0.589 Optimal trade-off; superior perceptual performance
Oracle \(t_{\text{GT}}\) 22.698 0.690 4.656 70.466 0.717 0.606 Theoretical upper bound using ground-truth reference

Table 2: Ablation on Noise Supervision and Distillation Schemes (Paper Tables 3 & 5)

Ablation Axis Specific Configuration LIQE ↑ MUSIQ ↑ CLIPIQA ↑ MANIQA ↑ Core Takeaway
Noise Supervision \(\mathcal{L}_\epsilon\) Without loss (W/O loss) 4.004 67.256 0.626 0.483 Noise deviates from Gaussian prior; sharp metric drop
Explicit KL divergence \(\mathcal{L}_{\text{KL}}\) 4.218 69.363 0.659 0.523 Enforcing uniform Gaussianity collapses sample diversity
Consistency Loss \(\mathcal{L}_\epsilon\) (Ours) 4.668 69.853 0.709 0.589 Maintains Gaussian validity with content-dependent noise
Score Distillation \(\mathcal{L}_{\text{distill}}\) Random \(t\) + Random noise 3.990 68.120 0.636 0.479 Conventional stochastic distillation suffers high variance
Predicted \(\hat{t}\) + Random noise 4.174 69.234 0.676 0.506 Accurate timestep markedly stabilizes score targets
Random \(t\) + Predicted \(\hat{\epsilon}\) 4.045 68.519 0.653 0.499 Noise prediction alone provides marginal benefit
Predicted \(\hat{t}\) + Predicted \(\hat{\epsilon}\) (Ours) 4.668 69.853 0.709 0.589 Full synergy unlocks maximum restoration quality

Key Findings

  1. Resolving the Fixed-Timestep Trade-off: Any fixed timestep inevitably suffers from the perceptual-distortion dilemma (low \(t\) preserves fidelity but lacks sharpness; high \(t\) generates details but induces structural distortion). In contrast, the adaptive timestep \(\hat{t}\) in IAC-IR dynamically hits the sweet spot, outperforming all fixed settings in perceptual quality (LIQE 4.668) while closely matching the oracle \(t_{\text{GT}}\) bound.
  2. Flexible Gaussianity Prevents Mode Collapse: Enforcing a strict KL divergence penalty on predicted noise forces the network to output homogeneous noise across all inputs (evidenced by tight clustering in t-SNE projections). Conversely, the proposed \(\mathcal{L}_\epsilon\) consistency loss and statistical moment objective \(\mathcal{G}(\cdot)\) accommodate spatially varying degradation characteristics while honoring Gaussian properties.
  3. Unmatched Operational Efficiency: Evaluated on an A100 GPU for \(512\times 512\) image restoration, IAC-IR runs at a latency of only 110ms with 1384 GMACs, achieving faster throughput than InvSR (117ms / 2123 GMACs) and HYPIR (220ms / 1784 GMACs), while operating over \(30\times\) faster than multi-step baseline StableSR (3460ms).

Highlights & Insights

  • Inverse Translation of Degradation to Diffusion Alignment: Instead of forcing the denoiser to adapt to non-standard LQ distributions, IAC-IR maps degraded inputs onto their exact corresponding states along the pretrained forward trajectory, fully unlocking the frozen generative prior.
  • Label-Free Indirect Corruption Supervision: By formulating a multi-moment Gaussian penalty \(\mathcal{G}(\cdot)\) on candidate residuals and coupling it with frozen denoiser reconstruction consistency, the method establishes rigorous supervision without requiring empirical timestep annotations.
  • Deterministic Alignment for Score Distillation: Replacing stochastic timesteps and noise in score distillation with deterministic, input-aligned corruption factors drastically suppresses gradient variance, establishing an effective design pattern for generative distillation.

Limitations & Future Work

  • Perception-Distortion Tension: While perceptual metrics are dominant, pixel-wise fidelity metrics like PSNR remain slightly lower than conservative regression-focused models. Future work could introduce a runtime controllability mechanism to allow interactive trade-off tuning via timestep modulation.
  • Global Scalar Timestep Assumption: Currently, \(\hat{t}\) is predicted as a single global scalar for the entire image. In scenarios with extreme non-uniform degradation (e.g., local defocus blur alongside sharp background elements), extending this to a spatially adaptive, patch-level timestep map warrants further investigation.
  • vs InvSR (CVPR 2024): InvSR pioneered noise inversion but kept the timestep fixed and lacked explicit supervision for noise estimation, assuming symmetric degradation across LQ and HQ. IAC-IR jointly predicts both adaptive timesteps and noise under rigorous self-supervision, eliminating severe failure modes on unscaled benchmarks like RealPhoto.
  • vs HYPIR (2024): HYPIR relies on heavy Transformer modules to capture degradation, incurring considerable computational latency (220ms) and struggling on sensor-shift benchmarks. IAC-IR introduces only a compact ViT estimator (2% parameters), halving latency while achieving broader generalization.
  • vs TSD-SR (CVPR 2025): While TSD-SR attempted to stabilize single-step distillation using auxiliary target scores, it still relied on heuristic corruption. IAC-IR solves the core bottleneck by purifying score gradients with deterministic, input-aware corruption factors.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant indirect self-supervision utilizing intrinsic diffusion statistics to learn input-adaptive corruption parameters.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 diverse benchmarks, extensive ablation variants, user studies, and detailed latency benchmarks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Methodological formulations and motivation are structured with high technical clarity and seamless narrative flow.
  • Value: ⭐⭐⭐⭐⭐ Resolves a fundamental challenge in one-step diffusion restoration, delivering state-of-the-art perceptual quality with practical real-time efficiency.