Skip to content

NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment

Conference: ECCV 2026
arXiv: 2606.18066
Code: None (The original paper does not provide an official code repository for NTRK, only linking to base models such as FLUX / Wan2.1)
Area: Diffusion Models / Alignment RLHF
Keywords: Noise-Tilted Reverse Kernels, Whitening Operator, Inference-Time Reward Alignment, Confidence Interval Projection, Training-Free Guidance

TL;DR

This paper proposes NTRK (Noise-Tilted Reverse Kernel): during diffusion inference, instead of altering the mean of the reverse kernel, it transforms the reward gradient into a valid Gaussian noise direction via a "whitening operator" and injects it into the noise term. This achieves both "gradient guidance" and "noise compatibility" while sampling only one candidate per step. In aesthetic image generation, with only 25 NFE, it outperforms the strongest baseline at 500 NFE, reducing the computational cost by 20 times.

Background & Motivation

Background: Inference-time reward alignment has become one of the most effective approaches for scaling up pre-trained diffusion models. Without retraining the model, by encoding user preferences into a reward function \(r(\cdot)\), the sampling distribution can be steered toward high-reward regions in each step of iterative denoising. This paradigm is widely used in deblurring, super-resolution, aesthetic generation, text-to-image alignment, and controllable video generation.

Limitations of Prior Work: All these approaches essentially tackle the same core question: how to modify or utilize the reverse Gaussian kernel to incorporate reward information. They fall into two mutually exclusive paradigms, each with prominent drawbacks. Mean-shifted (or gradient guidance) approaches, represented by DPS, directly add the reward gradient to the mean of the reverse kernel (see the equation \(\tilde{\bm{\mu}}_\theta=\bm{\mu}_\theta+\lambda_t\nabla_{\bm{x}_t}r(\hat{\bm{x}}_{0|t})\)). Although they leverage gradient directions, they push the intermediate states out of the "noise-compatible regime"โ€”the narrow manifold where the pre-trained model was actually trained. Larger gradients cause greater deviations, leading to out-of-distribution (OOD) degraded samples and reward hacking. On the other hand, search-based approaches like SVDD draw \(K\) candidates from the original reverse kernel and select one using reward argmax. While they fully preserve the native kernel and image quality, they lack any gradient signal, relying solely on random sampling to hit high-reward directions. This requires a massive number of samples when high-reward instances lie in low-density regions.

Key Challenge: Gradient guidance and noise compatibility represent the two ends of a trade-off. To the authors' knowledge, no prior method has achieved both simultaneously (see Table 1).

Goal / Key Insight: The authors aim to fill this gap. The key observation is that search-based methods preserve image quality because they leave the mean unaltered and only manipulate the noise term (choosing a good noise is essentially an implicit "noise tilting"). Is it possible to construct that optimal noise directly, rather than sampling \(K\) candidates?

Core Idea: Keep the pre-trained mean \(\bm{\mu}_\theta\) completely untouched, and inject all reward information through the noise term. The difficulty is that the reverse kernel is Gaussian, meaning the injected perturbation must remain a "typical Gaussian noise" (not only matching the norm but also being spatially uncorrelated and satisfying Gaussian statistics globally). However, the original reward gradient is structured and deterministic; directly injecting it as noise results in "atypical noise," which similarly drives states out of distribution (Figure 2). To address this, this paper introduces the concept of whitening from statistical learning and designs a whitening operator \(\mathcal{W}\) to process the gradient into a noise-compatible direction before injection. Whitening is not merely an auxiliary step but the core mechanism of NTRK, and the alignment performance directly scales with the quality of the whitening.

Method

Overall Architecture

NTRK addresses the problem of how to inject reward gradients into the reverse kernel without destroying the noise-compatible regime. The overall workflow consists of only two steps: (1) Use the whitening operator \(\mathcal{W}\) to transform the structured reward gradient \(\nabla_{\bm{x}_t}r(\hat{\bm{x}}_{0|t})\) into a valid white Gaussian noise direction \(\bm{w}_t\); (2) Combine \(\bm{w}_t\) and a separate independent random noise vector using the "Gaussian mixture identity" with weights \(\sqrt{\rho_t}\) and \(\sqrt{1-\rho_t}\) to form \(\tilde{\bm{\epsilon}}_t\). This replaces \(\bm{\epsilon}_t\) in the base sampling formulation while leaving the mean \(\bm{\mu}_\theta\) and the nominal noise scale \(\sigma_t\) completely unchanged. Since the core comprises a pure sampling kernel, a kernel derivation, and a projection-style whitening operator, it is explained through mathematical formulations below rather than a pipeline diagram.

To formalize, the reward alignment objective is to find a target distribution \(p_0^*=\arg\max_q \mathbb{E}_{\bm{x}_0\sim q}[r(\bm{x}_0)]-\beta\,\mathcal{D}_{\text{KL}}[q\|p_0]\) (where smaller \(\beta\) denotes stronger reward tilting). The corresponding optimal reverse kernel can be approximated as \(p_\theta^*(\bm{x}_{t-1}|\bm{x}_t)\propto p_\theta(\bm{x}_{t-1}|\bm{x}_t)\exp(V_{t-1}(\bm{x}_{t-1})/\beta)\), where the intractable value function is approximated using the Tweedie posterior mean \(V_t(\bm{x}_t)\approx r(\hat{\bm{x}}_{0|t})\) with \(\hat{\bm{x}}_{0|t}:=\mathbb{E}[\bm{x}_0\mid\bm{x}_t]\). The four types of reverse kernels are four implementations of this approximation, classified by whether the standardized perturbation \(\bm{\eta}_t:=(\bm{x}_{t-1}-\bm{\mu}_\theta)/\sigma_t\) remains standard Gaussian noise:

  • Base: \(\bm{x}_{t-1}=\bm{\mu}_\theta+\sigma_t\bm{\epsilon}_t\), where \(\bm{\eta}_t=\bm{\epsilon}_t\) is naturally Gaussian but contains no reward information.
  • Mean-Shifted (DPS): \(\bm{\eta}_t^{\text{mean}}=\bm{\epsilon}_t+\frac{\lambda_t}{\sigma_t}\nabla_{\bm{x}_t}r\), which includes an extra deterministic gradient term and is no longer Gaussian noiseโ€”this is the root cause of quality degradation and represents the key mismatch pointed out in this paper.
  • Search-Based (SVDD): \(K\) candidates are sampled to select the one with the highest reward \(\bm{\epsilon}_t^{(i^\star)}\). Here, \(\bm{\eta}_t\) remains Gaussian (since the selected candidate is still a true Gaussian sample), but at the cost of \(K\) evaluations per step.
  • Noise-Tilted (NTRK, Ours): \(\bm{\eta}_t^{\text{noise}}=\tilde{\bm{\epsilon}}_t\), requiring only a single sample while being both gradient-informed and a valid Gaussian direction.

Key Designs

1. Noise-Tilted Reverse Kernel: Shifting Reward from Mean to Noise to Maintain Validity via Gaussian Mixtures

The limitation of mean-shifting is that it contaminates \(\bm{\eta}_t\). The mechanism of NTRK is to keep \(\bm{\mu}_\theta\) unchanged and replace only the noise term. It leverages a fundamental identity: if \(\bm{\epsilon}_1,\bm{\epsilon}_2\sim\mathcal{N}(\bm{0},\bm{I})\) are independent, then for any \(\rho\in[0,1]\), we have \(\sqrt{\rho}\,\bm{\epsilon}_1+\sqrt{1-\rho}\,\bm{\epsilon}_2\sim\mathcal{N}(\bm{0},\bm{I})\). Thus, the guided noise is defined as:

\[\tilde{\bm{\epsilon}}_t=\sqrt{\rho_t}\,\bm{w}_t+\sqrt{1-\rho_t}\,\bm{\epsilon}_t,\qquad \bm{x}_{t-1}=\bm{\mu}_\theta(\bm{x}_t,t)+\sigma_t\tilde{\bm{\epsilon}}_t\]

where \(\bm{w}_t:=\mathcal{W}(\nabla_{\bm{x}_t}r(\hat{\bm{x}}_{0|t}))\) is the whitened reward direction, \(\bm{\epsilon}_t\) is an unbiased random term, and \(\rho_t\in[0,1]\) controls the guidance strength. This step is effective because as long as \(\bm{w}_t\) itself is a valid white noise vector, the mixture is guaranteed to strictly follow \(\mathcal{N}(\bm{0},\bm{I})\)โ€”which means \(\bm{\eta}_t^{\text{noise}}=\tilde{\bm{\epsilon}}_t\) remains standard Gaussian. The mean of the reverse kernel and \(\sigma_t\) match the base model, staying within the noise-compatible regime naturally. It is equivalent to compressing SVDDโ€™s "sample \(K\) and select" into "directly construct the best one," reducing the per-step cost from \(K\) to \(1\) (a heuristic explanation of \(\rho_t\) relative to search guidance under local linearity is provided in Appendix A.5 of the paper).

2. Whitening Operator \(\mathcal{W}\): Projecting Structured Gradients onto the "Typical Gaussian Set" without Losing Directionality

The validity of the previous step entirely depends on whether \(\bm{w}_t\) is a valid white noise vector. However, the original gradient is structured and deterministic, and directly injecting it results in atypical noise (as shown in Figure 2, atypical noise induces artifacts). The difficulty is that while \(\mathcal{N}(\bm{0},\bm{I})\) has density everywhere in high dimensions, almost all probability mass is concentrated in an extremely narrow shell called the typical set, which lacks a closed-form representation. \(\mathcal{W}\) addresses this by approximating the typical set using a family of high-confidence constraints derived from known standard normal statistics, and defining \(\mathcal{W}\) as a sequence of Euclidean projections onto the corresponding confidence sets (all taking a \(99.99\%\) confidence bound, \(\alpha=10^{-4}\)).

Its core building block is Two-level Order Statistics (2OS). The vector to be whitened, \(\bm{x}\in\mathbb{R}^N\), is reshaped into a tile matrix \(\bm{Y}\in\mathbb{R}^{M\times D}\) (\(N=MD\)), and then sorted along both rows and columns: \(\bm{Z}=\mathrm{sort}_0(\mathrm{sort}_1(\bm{Y}))\). Intuitively, \(Z_{r,j}\) is the "r-th smallest value among the j-th order statistics of all tiles," functioning as a "rank of ranks" summary. For standard Gaussian noise, each \(Z_{r,j}\) is sharply concentrated. Therefore, a tight quantile interval \((L_{r,j},U_{r,j})=(\Phi^{-1}(q_{r,j}^{\text{lo}}),\Phi^{-1}(q_{r,j}^{\text{hi}}))\) can be calculated for each \((r,j)\) using Beta quantiles of nested order statistics. The projection is performed via sort-clip-unsort: sorting \(\rightarrow\) clipping each \(Z_{r,j}\) to its interval \(\rightarrow\) restoring the original permutation. The authors show that this is equivalent to the Euclidean projection onto the confidence set \(\mathcal{C}_{\text{2os}}=\{\bm{Y}:L_{r,j}\le Z_{r,j}\le U_{r,j}\}\). Why it works: 2OS prevents extreme values from clustering within a few tiles and forces each tile to have a balanced distribution of values, matching true Gaussian behavior. Applying 2OS to tile-level mean/energy statistics constrains block-level moments, and repeating this across orthogonal transform domains (such as the Fourier domain) captures structural correlations that value-domain projections cannot reach. The complete \(\mathcal{W}\) is the composition of these 2OS projections across multiple domains.

3. Confidence Interval Projection vs. Hard Constraints: Only Suppressing Atypical Structures while Leaving True Noise Unchanged

This is the key improvement of \(\mathcal{W}\) over existing noise-compatibility methods. Previously, WGNC framed noise compatibility as a projection problem for the first time, but used hard equality norm constraints in the Fourier domain, which had the side effect of distorting even naturally valid noise. NTRK replaces hard equalities with confidence interval projections: structured inputs are heavily whitened, whereas typical Gaussian noise, which already falls within the \(99.99\%\) interval, passes through almost unaltered (as shown in the rightmost two columns of Figure 4, cosine similarity \(>0.99999\)). This property is crucialโ€”it ensures that the random term \(\sqrt{1-\rho_t}\,\bm{\epsilon}_t\) in the formulation is not corrupted by the operator, keeping the overall mixture as clean Gaussian noise. โš ๏ธ Note that the complete multi-domain construction of 2OS and Beta quantile derivations are in Appendix B of the paper.

Loss & Training

NTRK is a completely training-free, inference-time sampling method. It does not introduce any loss function or update any parameters; it serves as a drop-in replacement for existing sampling loops. Key hyperparameters include the guidance strength \(\rho_t\in[0,1]\), KL temperature \(\beta\), and a fixed confidence level of \(1-\alpha=99.99\%\). All experiments uniformly use 25 sampling steps. NTRK can be combined with Best-of-N (BoN), a multi-particle strategy where multiple independent sampling trajectories are run to select the one with the highest reward to match the total NFE budget. In the main text, methods using BoN are denoted with โ€ . Since it only modifies sampling and leaves weights untouched, it is orthogonal to fine-tuning (e.g., MixGRPO) and can be directly applied on top of fine-tuned models.

Key Experimental Results

Setup: Image tasks use FLUX and video tasks use Wan2.1 as base flow models, with 25 sampling steps. Single-particle methods (DPS/FreeDoM) are aligned to the same NFE using BoN for a fair comparison. The evaluation distinguishes between the target reward (seen during optimization) and the held-out reward (unseen).

Main Results

Aesthetic + Text-aligned Image Generation (selected from Table 2, โ†‘ is better, bold indicates the best in the column):

Method NFE Aesthetic-Aesthetic โ†‘ Aesthetic-HPSv2 โ†‘ Aesthetic-ImageReward โ†‘ Text To Image-PickScore โ†‘ Text To Image-Aesthetic โ†‘
Base [FLUX] 25 6.0282 0.2759 1.0538 0.2054 5.4664
BoN 500 6.7310 0.2890 1.1419 0.2146 5.8582
DPSโ€  500 6.7647 0.2861 1.0639 0.2147 5.8073
FreeDoMโ€  533 6.8406 0.2853 0.9941 0.2133 5.8492
SVDD 500 7.1363 0.2814 1.0256 0.2204 5.8743
RBF 500 6.9900 0.2826 1.0761 0.2202 5.8618
DAS 500 6.9384 0.2860 1.0568 0.2139 5.8385
ฮจ-Sampler 500 7.0116 0.2847 1.1235 0.2120 5.7329
NTRK (Ours) 25 7.4510 0.2928 1.2565 0.2224 5.7720
NTRKโ€  (Ours) 500 7.9656 0.2932 1.1669 0.2327 5.9020

Key takeaway: NTRK with only 25 NFE (1/20th of the computation) achieves an aesthetic score of 7.4510, already exceeding the highest score among all 500 NFE baselines (SVDD: 7.1363). Under 500 NFE with BoN, NTRK further improves to 7.9656. It also achieves the best performance on held-out metrics (ImageReward / HPSv2), demonstrating that the improvement is not achieved through reward hacking at the expense of image quality.

Video Generation (Table 3, target = VideoReward = sum of MQ+VQ+TA):

Method NFE VideoReward โ†‘ Dynamic โ†‘ Aesthetic โ†‘ Subject โ†‘
Base [Wan2.1] 25 -0.399 0.9300 0.6104 0.9589
DPS 25 -0.130 0.9350 0.5867 0.9398
FreeDoM 25 -0.211 0.6900 0.5880 0.9586
NTRK (Ours) 25 3.465 0.9500 0.6120 0.9591

The VideoReward score increases dramatically from -0.399 (Base) / -0.130 (DPS) to 3.465, which is a significant margin. It also achieves optimal or near-optimal results on multiple held-out VBench metrics.

On top of fine-tuned models (Table 4, base = FLUX + MixGRPO fine-tuned, target = PickScore):

Method NFE PickScore โ†‘ Aesthetic โ†‘ HPSv2 โ†‘ ImageReward โ†‘
Base 25 0.2054 5.4664 0.2316 0.1710
MixGRPO 25 0.2166 6.5245 0.2679 0.7605
โ”” DPSโ€  500 0.2235 6.6966 0.2840 1.0501
โ”” NTRKโ€  (Ours) 500 0.2545 6.7281 0.3224 1.2648

When applied to a fine-tuned model, NTRK consistently outperforms DPS on both target and all held-out metrics, confirming that "inference-time alignment is orthogonal to fine-tuning and can be stacked." โš ๏ธ Note that the Aesthetic score for NTRKโ€  in Table 4 is 6.7281 in the original paper, which is recorded here. Refer to the original paper for accuracy.

Ablation Study

The paper does not include a traditional component-by-component ablation table. Its core ablation is reflected in the NFE-efficiency dimension of Table 2, where results are presented for both 25 NFE and 500 NFE to disentangle the contribution of "the method itself" from "BoN multi-particle sampling." The fact that NTRK at 25 NFE (pure method, without BoN) already outperforms all 500 NFE baselines shows that the gains primarily stem from the noise-tilted kernel rather than sampling scaling. Additionally, Figure 4 provides a qualitative component-wise ablation of the whitening operator \(\mathcal{W}\): incrementally applying the 2OS projection \(\rightarrow\) tile-level statistics \(\rightarrow\) multi-domain transformations makes the latent space resemble typical Gaussian noise more closely, producing increasingly realistic samples. When typical Gaussian noise is input, \(\mathcal{W}\) leaves it virtually unchanged (cosine similarity \(>0.99999\)), demonstrating the design goal of the confidence interval projection compared to WGNC's hard constraints: "constraining only structural features without degrading true noise."

Key Findings

  • The most significant contribution comes from the whitening operator \(\mathcal{W}\): The authors explicitly note that progress in alignment directly correlates with whitening quality, making \(\mathcal{W}\) the core enabler for the "noise tilting" concept. Removing whitening and directly injecting gradients degrades the output into atypical noise, inducing artifacts (Figure 2).
  • Outstanding efficiency gains: Under aesthetic generation, 25 NFE matches or surpasses the strongest 500 NFE baseline, yielding a 20ร— reduction in computational cost. It replaces the process of "searching for \(K\) good noises" with "constructing 1 optimal noise," turning random search into deterministic guidance.
  • Robust held-out rewards: While many gradient-guided methods degrade held-out image quality when optimizing the target reward (reward hacking), NTRK consistently maintains state-of-the-art results on held-out metrics. This indicates actual alignment rather than overfitting.
  • Significant performance scale-up in video tasks: The VideoReward jumping from negative values to 3.465 is the most dramatic improvement in the paper. This suggests that in video domains where the pre-trained distribution is more fragile, the benefit of staying within the distribution is even more pronounced.

Highlights & Insights

  • Shifting rewards from the mean to the noise offers a refreshing perspective: While prior works struggle with modifying the mean, this paper takes the opposite route: leaving the mean completely untouched and packing all reward information into the noise term. This step ensures that "gradient guidance" and "staying within the training distribution" are no longer mutually exclusive (with NTRK being the only checkmarks in all three categories in Table 1).
  • Clever use of the Gaussian mixture identity: The identity \(\sqrt{\rho}\bm{\epsilon}_1+\sqrt{1-\rho}\bm{\epsilon}_2 \sim \mathcal{N}(\bm{0},\bm{I})\) is a standard textbook fact. However, using it to safely inject a guidance direction into the noise without breaking Gaussianity is an elegant design choice. It naturally provides a guidance strength control knob \(\rho_t\) with zero theoretical distribution shift.
  • Equivalence of sort-clip-unsort in 2OS to Euclidean projection: Proving that an intuitive sort-and-clip heuristic corresponds to the exact Euclidean projection onto a confidence set is mathematically elegant and reusable. This idea of "approximating high-dimensional typical sets using high-confidence intervals" can be extended to other scenarios requiring a vector to resemble white noise (e.g., noise optimization, diffusion guidance stabilization).
  • Confidence interval projection > hard constraints is a highly practical trick: While WGNC's hard equality constraints alter valid noise, switching to a \(99.99\%\) interval ensures that "only structured features are suppressed, and clean noise is preserved." The near-identity cosine similarity \(>0.99999\) makes it safe to integrate into sampling loops.
  • Orthogonal and stackable with fine-tuning: The premise that "parameter fine-tuning is orthogonal to sampling guidance" is validated in Table 4, implying that NTRK can serve as a free improvement layer for any already-aligned model.

Limitations & Future Work

  • Computational and engineering costs of the whitening operator are under-discussed: Performing multiple "sort-clip-unsort" operations across multiple transform domains at each step introduces overhead. The main text does not detail the wall-clock time or GPU memory consumption of \(\mathcal{W}\) compared to naive gradient guidance, reporting only NFEs. Sorting costs may be non-negligible on high-resolution images or long-video latents. โš ๏ธ The appendix may contain more details; refer to the original paper.
  • Hyperparameter \(\rho_t\) selection relies on heuristics: The mapping of \(\rho_t\) to search-based guidance is conceptually explained under a local linearity assumption (Appendix A.5), which is not strictly optimal. There is a lack of systematic guidance on how to schedule \(\rho_t\) across different tasks or timesteps.
  • Typical set approximation is a "constraint approximation" rather than an exact representation: Since \(\mathcal{W}\) approximates the typical set with a family of confidence constraints, it cannot formally guarantee coverage of all properties of the typical set, potentially allowing residual high-order structural correlations to slip through.
  • Lack of traditional quantitative ablation: There is no quantitative table evaluating individual component contributions (e.g., removing 2OS, multi-domain, or tile statistics), only qualitative progressions in Figure 4. This makes it difficult to measure the marginal contribution of each component.
  • Future Work: Extending confidence interval projections to more transform domains or learnable whitening; designing adaptive \(\rho_t\) schedules over time or rewards; exploring noise tilting on discrete diffusion or other modalities.
  • vs. DPS (Mean-Shifted): DPS adds the reward gradient to the reverse kernel mean to steer the diffusion process. However, the standardized perturbation \(\bm{\eta}_t\) includes a deterministic term, breaking noise compatibility and degrading image quality. NTRK keeps the mean unchanged and injects the whitened gradient into the noise term, maintaining a valid Gaussian \(\bm{\eta}_t\). The difference lies in "injecting rewards via the mean vs. via the noise."
  • vs. SVDD (Search-Based): SVDD draws \(K\) candidates at each step and selects the one with the highest reward. This preserves image quality but lacks gradient signals and scales linearly with \(K\). NTRK uses whitening to directly construct the optimal noise, achieving single-pass sampling and making implicit noise tilting explicit.
  • vs. DAS / ฮจ-Sampler (Hybrid/SMC): These methods integrate gradient guidance into SMC-like particle filters (ฮจ-Sampler also reshapes the initial particle distribution). However, they still rely on shifting the mean to inject gradients, thereby inheriting the noise-compatibility issues of mean-shifting. NTRK avoids modifying the mean by design.
  • vs. FreeDoM: FreeDoM applies additional MCMC sampling on top of mean-shifting to improve precision, which is computationally expensive (running only 13 steps in video tasks to match NFE budgets in the paperโ€™s experiments). NTRK requires only a single sample per step, offering significantly higher efficiency.
  • vs. WGNC (Noise-Compatible Projection): WGNC was the first to frame noise compatibility as a hard projector in the Fourier domain, but its hard constraints distort naturally valid noise. NTRKโ€™s \(\mathcal{W}\) improves upon this via confidence interval projection: "constraining only structured features and leaving clean noise untouched."
  • vs. DNO / MPGR / StressDream (Regularization): These works use soft constraints (such as norm or higher-order spectral statistics) to encourage Gaussian typicality, which cannot guarantee typicality. NTRK enforces typicality deterministically in a single pass via projection.

Rating

  • Novelty: โญโญโญโญโญ The combination of "rewards via noise instead of mean" and "sort-clip-unsort as Euclidean projection" elegantly fills the gap between gradient guidance and noise compatibility with novel perspectives and mechanisms.
  • Experimental Thoroughness: โญโญโญโญ Broad coverage across aesthetics, text-to-image, video, and fine-tuning integration, evaluating both target and held-out scores, though lacking a quantitative ablation of whitening components and computational overhead analysis.
  • Writing Quality: โญโญโญโญ The classification of the four kernels based on the Gaussianity of the standardized perturbation is highly clear. Table 1 and Figure 3 are extremely effective. Whitening details are deferred to the appendix, which sometimes requires back-and-forth reading.
  • Value: โญโญโญโญโญ Drop-in, training-free, orthogonal to fine-tuning, and offering a 20ร— efficiency speedup. It can be easily stacked onto any pre-trained or aligned diffusion model, possessing high practical value.