Adaptive Noise Covariance Scheduling under Riemannian Metrics for Diffusion Models¶

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/BolinDeng/RMDM
Area: Image Generation / Diffusion Models
Keywords: blue noise, covariance geodesics, adaptive scheduling, spectral evolution, deterministic diffusion

TL;DR¶

The paper separates blue-to-white noise evolution into a covariance path and a per-image transition speed, combining a Bures-Wasserstein geodesic with a latent-predicted temperature to improve image-generation FID and KID without replacing the BNDM denoising backbone.

Background & Motivation¶

Reverse diffusion typically builds low-frequency structure before recovering high-frequency detail, whereas white noise has a flat power spectrum. BNDM already addresses this mismatch by transitioning from high-frequency-emphasizing blue noise to white noise, allowing low-noise stages to perturb texture while retaining coarse structure. However, matching the two endpoints does not determine the intermediate noise statistics. BNDM interpolates Cholesky factors of the endpoint covariances, and the resulting intermediate covariances can exhibit directional spectral artifacts and substantial average-power fluctuations.

A second issue is the fixed transition schedule. Fur, faces, buildings, and mixed animal categories have different frequency characteristics, so a globally shared schedule may be unsuitable. Simply learning a more flexible schedule is not sufficient either: different covariance paths respond differently to schedule changes. In the paper's ablation, making the LE/AIRM schedule adaptive actually worsens performance. The relevant design problem is therefore the interaction between path geometry and controllability, not merely the number of learnable parameters.

Core idea: choose a BW geodesic with smooth spectral evolution and a coloring operator that responds uniformly to schedule changes, then predict a transition temperature from an image-associated latent variable so that path shape and traversal speed can be studied separately.

Method¶

Overall Architecture¶

The method uses BNDM as its host, retaining the U-Net, deterministic forward blending, and two-output reverse update. Offline, it computes a symmetric matrix square root of the existing blue-noise covariance. During training, a clean image is encoded into a latent variable that predicts a schedule temperature; this temperature determines positions along the BW path, whose coloring operators construct noisy training images. The denoiser, encoder, and temperature predictor are trained jointly.

At generation time, there is no clean image to encode, so the initial image noise and schedule latent are sampled independently from standard Gaussian priors. The temperature is predicted once and held fixed throughout that sample's trajectory; the schedule value changes with time. This is not a procedure that repeatedly inspects the evolving image and updates its temperature at every step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Blue-noise covariance<br/>Offline eigendecomposition"] --> B["BW Covariance Path"]
    B --> C["Latent-Adaptive Schedule"]
    D["Training: image posterior<br/>Generation: Gaussian prior"] --> C
    C --> E["Two-Head Denoising<br/>and Joint Training"]
    F["Image and white noise<br/>Forward blending in training"] --> E
    E --> G["Reverse iterations<br/>Generated image"]

Key Designs¶

1. BW Covariance Path: replace factor interpolation to control intermediate noise statistics

Let the blue-noise covariance be \(\Sigma_b\) and the white-noise covariance be the identity. BNDM linearly interpolates a Cholesky factor with the identity and applies that operator to white noise. The proposed method instead uses the symmetric square root of the blue-noise covariance, computed offline from eigenvectors \(Q\) and a diagonal matrix of positive eigenvalues \(\Lambda\). For the specific blue-to-white endpoints, the BW coloring operator is particularly simple, as given in Equation (9) and Algorithm 1:

\[ C^{\mathrm{BW}}_{\gamma}=(1-\gamma)Q\Lambda^{1/2}Q^\top+\gamma I, \qquad \Sigma^{\mathrm{BW}}_{\gamma}=C^{\mathrm{BW}}_{\gamma}(C^{\mathrm{BW}}_{\gamma})^\top. \]

Here, \(\gamma\) controls the blue-to-white transition, not the overall corruption strength. Zero gives blue noise, one gives white noise, and intermediate positions preserve the covariance eigenvectors while changing eigenvalues. For the paper's blue-noise endpoint, the central low-frequency hole gradually fills instead of developing the oblique artifacts observed with Cholesky interpolation. This should not be generalized into a claim that any covariance with fixed eigenvectors is automatically isotropic; the evidence concerns the particular blue-noise construction used here.

The derivative of the BW coloring operator with respect to \(\gamma\) is constant: the identity minus the symmetric blue-noise square root. Equal schedule increments therefore induce equal operator increments throughout the path. Cholesky interpolation also has constant sensitivity, but its spectral directions drift. LE and AIRM coincide for the white-noise endpoint and preserve eigenvectors, yet their coloring operators respond nonlinearly to schedule position. BW is useful because it combines these properties, not because it is the only geodesic option.

The paper measures average noise power as the mean marginal variance and compares its cumulative absolute change over time in Figure 1:

\[ \delta_m^2=\frac{1}{n}\operatorname{tr}(\Sigma_{\gamma_t}). \]

Here, \(n\) is the covariance dimension. The observed lower fluctuation along the BW path motivates the claim that adjacent denoising targets change more gently in their statistics. This is analytical and empirical support for the chosen endpoints and schedules, not a universal optimality guarantee across arbitrary distributions or metrics.

2. Latent-Adaptive Schedule: use one temperature to control each trajectory's transition

Once the path is fixed, the model still needs to determine where to be at each timestep. The method retains a normalized sigmoid family: normalized time is mapped into an interval defined by start and end parameters, scaled by a temperature, and passed through a sigmoid. Subtracting the initial value and dividing by the endpoint difference makes the schedule increase from zero to one. The interval endpoints stay fixed; only the positive temperature varies by sample. The learnable control is consequently low-dimensional, rather than an arbitrary covariance matrix or timestep sequence.

During training, a lightweight CNN followed by three residual blocks predicts the mean and variance of a Gaussian posterior associated with the clean image. A reparameterized latent sample passes through a two-layer MLP to obtain the temperature. KL regularization toward a standard Gaussian prior makes it possible to sample schedules without an image at generation time. In unconditional generation, this latent controls the schedule; it is not decoded into an image by a VAE decoder, nor is it demonstrated to be an interpretable semantic control. The cached main text requires a positive temperature but does not specify the output transformation enforcing positivity, so no particular activation should be invented.

This arrangement connects image-informed training schedules to generation without clean-image input. For super-resolution, the paper separately predicts the temperature from the low-resolution input and concatenates that input with the noisy image along the channel dimension. This conditional extension should not be confused with unconditional prior sampling.

3. Two-Head Denoising and Joint Training: compensate for both blending and noise-color changes

This is an inherited BNDM host mechanism, not a newly invented network architecture. Two distinct time-dependent quantities matter: the overall mixing coefficient \(\alpha_t=t/T\) controls the proportion of clean image and noise, while the adaptive schedule \(\gamma_t^a\) determines the noise color. Algorithm 1 constructs the forward sample as:

\[ x_t=\alpha_t C^{\mathrm{BW}}_{\gamma_t^a}\epsilon+(1-\alpha_t)x_0, \qquad \epsilon\sim\mathcal N(0,I). \]

At the initial endpoint, the mixing coefficient is zero and the input is the clean image. At the terminal endpoint, both the mixing coefficient and blue-to-white schedule equal one, leaving white noise. Image-dependent intermediate paths therefore do not change the terminal Gaussian prior. The same underlying white-noise realization is transformed by the time-dependent coloring operators; this should not be described as independently drawing fresh noise at every forward step as in a DDPM-style explanation.

In reverse generation, a single direction saying how much noise to remove does not capture both changing mixture weights and changing noise color. The U-Net output is split into two parts: one predicts the difference between the clean image and current colored noise, while the other predicts the correction associated with covariance-path evolution. The sampler weights the two outputs by adjacent-step differences in the mixing coefficient and schedule, respectively, and updates the image iteratively. The second term accounts for the changing path rather than generating a separate type of visual content.

Loss & Training¶

Equation (27) combines the two prediction errors with posterior KL regularization, weighting the second-head error by the ratio of the schedule increment to the mixing increment. The denoiser, posterior encoder, and temperature MLP are optimized jointly. Sampling removes the encoder and retains the prior latent, MLP, and denoiser.

The cached extraction damages exponents and layout in Equations (24)-(27), particularly the matrix power in the second-head target. This note therefore does not transcribe that target's full formula or provide an executable reconstruction. Algorithms 1 and 2 still establish the forward construction and two-head update described above; exact reproduction requires checking the original PDF or author code.

Training uses 1000 timesteps and standard evaluation uses 250 sampling steps, with schedule endpoints set to 0 and 3. The denoiser learning rate is \(10^{-4}\), the encoder and MLP learning rate is \(10^{-5}\), the KL weight is 1, and optimization uses AdamW. The blue-noise covariance comes from offline statistics, and its symmetric square root is also computed offline. A full eigendecomposition should therefore not be counted as a per-step online training operation.

Key Experimental Results¶

Main Results¶

Experiments study pixel-space unconditional generation using Inception-v3 features. Lower FID and KID are better; generative Precision measures fidelity and Recall measures coverage of the real distribution. The following selection from Table 1 spans datasets and resolutions. The final column is the absolute difference between BNDM FID and the proposed method's FID.

Dataset / Resolution	IADB FID	BNDM FID	Ours FID	FID Reduction
AFHQ-Cat, 64×64	9.18	7.95	6.17	1.78
AFHQ-Cat, 128×128	10.81	9.47	8.47	1.00
CelebA, 64×64	7.53	7.17	6.19	0.98
LSUN-Church, 64×64	9.95	11.35	8.26	3.09
AFHQ, 64×64	11.06	12.79	9.51	3.28
AFHQ-Cat, 256×256	12.56	13.50	8.90	4.60
CelebA, 256×256	19.25	16.11	12.14	3.97

These results support consistent FID gains for the tested hosts and datasets, not superiority over modern large-scale text-to-image systems. BNDM uses its original schedules without additional tuning. The paper places additional schedule-sensitivity tests in supplementary material that is not included in the current cache.

Ablation Study¶

Table 6 crosses path and schedule choices on AFHQ-Cat. CH denotes Cholesky interpolation, LA the shared LE/AIRM geodesic, and BW the proposed path. F is a shared fixed schedule and A is the adaptive schedule.

Path + Schedule	FID ↓	KID ↓	Precision ↑	Recall ↑
CH + F	7.95	0.0052	0.73	0.49
LA + F	7.95	0.0047	0.72	0.43
BW + F	6.40	0.0034	0.76	0.49
CH + A	6.43	0.0037	0.78	0.44
LA + A	8.36	0.0053	0.69	0.44
BW + A	6.17	0.0034	0.79	0.45

Key Findings¶

With the fixed schedule, replacing CH with BW reduces FID from 7.95 to 6.40; adaptation then reduces it to 6.17. Path replacement yields the larger absolute improvement in this comparison, but component interactions prevent treating their benefits as independently additive.
Adaptation is not universally helpful: LA worsens from 7.95 to 8.36. For BW, reported KID stays at 0.0034, Precision rises from 0.76 to 0.79, and Recall drops from 0.49 to 0.45, exposing a fidelity-coverage trade-off.
In Table 2, the proposed method obtains AFHQ-Cat FID 6.74 with 50 steps, outperforming BNDM's 7.95 with 250 steps. Its five-step FID is still 57.35, however, so this is not evidence of high-quality extremely-few-step generation.
Extensions include a DDIM variant with FID 6.33, versus 9.82 for original DDIM and 7.11 for BNDM-DDIM. AFHQ-Cat super-resolution from 64×64 to 128×128 obtains PSNR 30.01 dB, compared with BNDM's 29.30 dB.
On CelebA-256, Table 5 reports total training computation of 202.56 GFLOPs versus 198.75 for BNDM, and 6.2 versus 6.0 hours for 100 epochs. The text reports 0.98M additional parameters; the table rounds total counts to 119.9M and 118.9M.

Highlights & Insights¶

Crossing path and schedule choices is more informative than reporting only the final model. The degradation of adaptive LA directly shows why the response of the path to its control variable matters before learning that control.
BW does not require expensive manifold optimization at every step here. Exploiting the identity white-noise endpoint reduces runtime construction to linear combinations involving a precomputed square root, making the geometric choice compatible with an existing training framework.
KL regularization addresses the mismatch between image-informed training and generation without clean-image input. It supplies a usable schedule distribution, but does not itself prove interpretable frequency semantics in the latent space.

Limitations & Future Work¶

Evaluation primarily covers faces, animals, and churches in pixel space, with reported resolutions up to 256×256. Large-scale latent text-to-image generation, video, and complex compositional conditioning remain untested.
The fixed blue-noise endpoint is inherited from the host. Adaptation changes the transition temperature, not the complete covariance for each image; endpoint adaptation might help more complex spectra but would introduce computational and stability challenges.
Main results and ablations do not report repeated-run variance or confidence intervals. In particular, the statistical stability of the 0.23 FID improvement from BW + F to BW + A cannot be determined from the table alone.
FID improvement should not obscure the decrease in Recall. Mode-coverage analysis, distributions generated at different temperatures, and KL-weight ablations would help test whether adaptation sacrifices rare modes.
The current cache contains only the main paper, not the cited supplementary derivations, sensitivity tests, or additional implementation details. Damaged formula extraction also limits equation-level reproduction. These research limitations are this note's assessment, not an attributed standalone limitations statement from the authors.

BNDM / Blue Noise for Diffusion Models already introduced blue-to-white noise transitions and supplies the host framework. The contribution here is a BW path based on the symmetric square root instead of a Cholesky path, followed by sample-dependent scheduling; blue-noise diffusion itself is not new to this paper.
IADB constructs a generative process through deterministic blending of images and noise. This work retains that interpretable construction while changing noise statistics over time, so its reverse update must account for both blending and color changes.
IHDM / Blurring Diffusion Models connect generation to frequency structure through heat dissipation or blurring. The present method instead chooses a geometric path in Gaussian noise covariance space, rather than directly designing an image-blurring operator.
DDIM receives a preliminary extension, suggesting that the idea is not strictly confined to BNDM. The evidence does not establish a training-free, plug-and-play benefit for arbitrary samplers or pretrained models.

Rating¶

Novelty: 4/5. Separating a BW covariance path from adaptive traversal speed is a clear mechanism, although it builds on existing blue-noise diffusion.
Experimental Thoroughness: 3/5. Multiple datasets, crossed path-schedule ablations, efficiency, and conditional generation are covered; broader model scales and statistical stability remain open.
Writing Quality: 4/5. Motivation, path properties, and ablations align well; damaged cached equations are an extraction issue rather than direct evidence of poor paper writing.
Value: 4/5. The geometric and control-oriented view of forward noise is reusable, with practical value bounded by the coverage trade-off and tested host settings.