Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/ShunqiM/Ctrl-Z-Sampling
Area: Image Generation
Keywords: diffusion models, inference-time compute, reward-plateau detection, adaptive rollback, candidate search

TL;DR

Ctrl-Z Sampling responds to stalled diffusion reward scores by randomly returning to noisier states and progressively deepening its search, matching or exceeding fixed-depth search on several quality measures with fewer denoiser calls, although not every metric improves.

Background & Motivation

Text-to-image diffusion models typically establish a coarse composition from Gaussian noise before refining textures and details. An image can consequently become sharper while still misrepresenting object counts, relative positions, or attribute assignments in the prompt. Once an incorrect low-frequency structure has formed early, later denoising often makes that mistake more complete rather than correcting it. Classifier-free guidance (CFG) strengthens the conditioning signal but does not guarantee escape from an established incorrect layout. Additional inference compute can therefore be spent on revising the trajectory, not just on adding ordinary denoising steps.

Existing Resampling uses shallow random re-noising to explore nearby states, Z-Sampling alternates unconditional inversion with conditional denoising, and Search-over-Path (SOP) compares multiple candidates. These methods demonstrate the value of going backward, but a fixed perturbation depth may repeatedly revisit the neighborhood of the same incorrect composition. Searching at every step also spends compute on trajectories that are still improving naturally. The paper describes this difficulty as a local optimum in surrogate quality space: a sample fails to obtain better preference scores within a bounded exploration region. Quality here is not the diffusion model's likelihood, and the hill-climbing analogy does not imply gradient optimization of a known true quality function.

The authors consequently connect when to explore with how far to roll back: a reward model first checks whether the predicted clean image keeps improving, and exploration is paid for only upon stagnation. When shallow rollback fails to find a sufficiently good continuation, the sampler progressively revisits noisier states where global structure can change more substantially. This does not promise a global optimum; it prioritizes correcting early mistakes over uniformly adding shallow trials under limited compute. Core Idea: trigger search from reward stagnation, create alternative trajectories through random re-noising, and adapt rollback depth using candidate feedback while global structure remains revisable.

Method

Overall Architecture

Inputs comprise the text condition, initial Gaussian noise, a frozen diffusion denoiser, a DDIM schedule, and an off-the-shelf reward model. The evolving state is a noisy latent, whereas the reward model evaluates a predicted clean image rather than scoring that noisy latent directly. For latent diffusion, the image-scoring interface needs a corresponding evaluable image representation; a latent tensor should not simply be treated as an RGB image. Ordinary steps still follow conditional DDIM; the contribution changes the search control between those steps rather than the denoiser architecture.

The process follows Reward-Plateau Detection, Controlled Random Rollback, and Candidate Selection and Depth Escalation. The first two determine when and where to explore again, while the last compares candidates at a common timestep and controls further search and state replacement. Exploration is enabled only during the first \(\lambda\) sampling steps; afterward, ordinary denoising completes the selected trajectory and produces the final image. All loops in the diagram are inference data flow, with no training supervision or parameter updates.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Text condition and noise"] --> STEP["Conditional DDIM step<br/>and clean-image prediction"]
    STEP --> DETECT["Reward-Plateau Detection"]
    DETECT -->|Stalled within window| REWIND["Controlled Random Rollback"]
    REWIND --> SELECT["Candidate Selection and<br/>Depth Escalation"]
    SELECT -->|Below threshold and depth available| REWIND
    SELECT -->|Use retained state| NEXT["Further denoising or final image"]
    DETECT -->|Progress or outside window| NEXT
    NEXT -->|Sampling steps remain| STEP

Key Designs

1. Reward-Plateau Detection: decide whether direct progress remains worthwhile

An ordinary DDIM update also provides a clean-sample estimate, which the authors reuse as an inexpensive proxy for the eventual output. The reward model \(R\) evaluates this estimate with the text condition \(c\) to obtain the current score \(r\). Scoring before completion allows search to intervene before the composition becomes difficult to change. This rests on an assumption: clean predictions at high noise levels must contain enough information for scoring to guide eventual quality. The paper supports this approach empirically but does not prove that intermediate scores agree strictly with final human preferences.

The branch condition in Algorithm 1 on page 7 can be expressed clearly as:

\[ r = R(c,\hat{x}_0^{\,t-1}),\qquad \operatorname{explore\ if}\quad r < r_{\mathrm{prev}} + \delta. \]

\(r_{\mathrm{prev}}\) is the reference score updated after the previously retained step and starts at \(-\infty\); the default threshold is \(\delta=0\). When the threshold is met, the ordinary branch retains the DDIM update and refreshes the reference without generating extra candidates. A positive threshold demands visible improvement before intervention stops, but can also repeatedly trigger expensive searches on an already strong high-score plateau. The paper leaves relative thresholds and global reward scheduling to future work; the present method uses a fixed offset.

2. Controlled Random Rollback: reopen structural choices from the current state

Upon stagnation, the algorithm starts from the current \(x_t\), rather than restarting from a finished image or simply retrieving an old historical latent. Using the original diffusion noise schedule, it attenuates the current signal and mixes in independent Gaussian noise to construct a higher-noise state. Each noise vector creates a new random continuation, making this controlled re-noising rather than an exact inverse of the deterministic DDIM trajectory. Shallow rollback retains more existing structure; deeper rollback allows larger changes but requires paying for denoising again. Depth denotes a discrete index span in the DDIM schedule, not an arbitrary noise variance specified directly.

Algorithm 1 caps the actual span at the initial timestep boundary:

\[ \Delta = \min(\text{inversion\_step},\,T-t). \]

Starting from the generated \(x_{t+\Delta}\), the sampler uses the same text condition and denoises for \(k=t+\Delta,\ldots,t\), returning to level \(t-1\). This compares alternative trajectories at the same progress level instead of directly ranking latents with different noise levels. Each candidate requires \(\Delta+1\) such denoising updates, so increasing depth is not a free random perturbation. Equations (2) through (4) on page 5 have missing terms and merged symbols in the text extraction; this note does not guess their exact original forms. The operational relationships above are supported by the prose and readable pseudocode; exact noise-schedule implementation should still be checked against the original equations or code.

3. Candidate Selection and Depth Escalation: try nearby continuations before widening the search radius

At each depth, \(N\) candidates use different Gaussian noise vectors, return to \(t-1\), and receive scores for their clean-image estimates. Search initializes its best state and score from the ordinary DDIM update, so it need not adopt any new candidate. A candidate replaces the retained state only when its score exceeds the current best; the algorithm does not blindly accept the last exploration result. Once the best score reaches the previous reference plus \(\delta\), search stops early without trying deeper inversions. Otherwise, rollback depth increases incrementally until the maximum depth \(d_{\max}\) is reached.

The early-exit threshold must be distinguished from the fallback choice after search exhaustion. Algorithm 1 still adopts the best observed state and updates the reference to its score after exhaustion, even when that score did not meet the old reference threshold. Its fallback is therefore no worse than the ordinary forward step included in the current search, but it does not establish strictly nondecreasing rewards across timesteps. This is more conditional than the prose summary that a zero threshold enforces nondecreasing reward; reproduction should follow the actual branch behavior. Candidates are independent before selection and can be evaluated in parallel, whereas successive depths depend on feedback.

A Worked Example

Consider a text prompt requiring a correct spatial relationship; the following illustrates state transitions rather than introducing a new quantitative experiment. With default \(T=50\), \(\lambda=40\), \(N=4\), \(d_{\max}=3\), and \(\delta=0\), the exploration window is \(t=50,\ldots,11\). At \(t=30\), the ordinary update first produces \(x_{29}\) and a corresponding clean prediction; if its reward reaches the reference, sampling proceeds directly. If reward decreases, the sampler first re-noises \(x_{30}\) to \(x_{31}\) and runs denoising along \(31\to30\to29\) for each of 4 candidates. Suppose none meets the threshold; it then tries deeper continuations from \(x_{32}\) back to \(x_{29}\). If depth 2 finds a best candidate that meets the threshold, that candidate is retained without spending the depth 3 budget. If all searches through depth 3 fail, it retains the highest-scoring state observed in this round, including the original ordinary \(x_{29}\). At \(t=10\) and below, this search is disabled and the model completes details along the selected trajectory.

Loss & Training

The method adds no loss, does not fine-tune the denoiser, and does not back-propagate reward gradients to optimize noise. ImageReward ranks candidates by default, and the CFG scale is 5.5; these are inference settings. The main table denotes the default configuration above as Ctrl-Z‡ and the lower-cost configuration with \(\lambda=30\), \(N=2\), and \(d_{\max}=3\) as Ctrl-Z†. Reported NFEs are the average denoiser calls per base denoising step: DDIM is marked 1.00, not one network call for an entire image. Cost depends on the number of stagnation events, candidate count, and realized rollback depths, so identical parameters can yield different NFEs across backbones or prompt collections. Reward-model passes, image decoding, and parallel scheduling costs cannot be read directly from that measure; NFEs are not end-to-end latency.

Key Experimental Results

Main Results

The paper evaluates Stable Diffusion 2.1 and Hunyuan-DiT on Pick-a-Pic, DrawBench, and T2I-CompBench. The table below selects SD2.1 results from Table 1 on page 10: 50 steps, CFG 5.5, and higher is better for HPSv2, AES, PickScore, and ImageReward (IR), while lower is better for NFEs. The two Ctrl-Z configurations are specified above; these comparisons use nearby, not exactly equal, NFE budgets.

Method	Pick-a-Pic HPSv2	Pick-a-Pic AES	Pick-a-Pic PickScore	Pick-a-Pic IR	DrawBench IR	NFEs
DDIM	25.34	5.649	20.67	0.194	0.046	1.00
SOP-1	26.34	5.684	20.85	0.735	0.637	3.00
Ctrl-Z†	26.44	5.686	20.88	0.720	0.650	2.77
SOP-4	27.23	5.700	21.12	1.113	1.008	9.00
Ctrl-Z‡	27.34	5.705	21.02	1.138	1.025	7.72

Ctrl-Z‡ reaches Pick-a-Pic IR 1.138 versus SOP-4's 1.113 while reducing NFEs from 9.00 to 7.72, differences of 0.025 and 1.28, respectively. However, its PickScore is 21.02 versus SOP-4's 21.12, and low-budget Ctrl-Z† also scores below SOP-1 on IR for this dataset. The results therefore support a favorable trade-off on several metrics, not a claim that every metric exceeds SOP.

The following table selects SD2.1 compositional-generation results from Table 2 on the same page, retaining the original numerical scales; higher is better in every column. Color, Shape, and Texture evaluate attribute binding, while Spatial and Numeracy concern spatial relationships and numerical constraints.

Method	Color	Shape	Texture	Spatial	Numeracy
DDIM	46.27	41.01	46.06	13.80	46.44
SOP-4	59.64	48.91	60.56	16.97	52.85
Ctrl-Z†	58.65	47.10	57.75	18.55	51.83
Ctrl-Z‡	61.26	53.97	62.24	19.29	53.73

Default Ctrl-Z improves Shape by 5.06 and Spatial by 2.32 over SOP-4, supporting the interpretation that rollback can improve some layout and compositional constraints. However, the same original table reports Hunyuan-DiT Numeracy of 56.69 for Ctrl-Z‡ versus 57.58 for SOP-4, so superiority is not universal across backbones either.

Ablation Study

Table 3 on page 13 compares exploration triggers with SD2.1 and ImageReward, keeping the remaining default settings. Always triggers unconditionally, Random triggers with \(p=0.5\), and Reward-Based uses the proposed stagnation rule; the table below selects a subset of the reported metrics.

Trigger	Pick-a-Pic HPSv2	Pick-a-Pic IR	DrawBench HPSv2	DrawBench IR	NFEs
Always (\(p=1.0\))	27.86	1.317	27.17	1.174	16.72
Random (\(p=0.5\))	27.00	1.062	26.71	0.906	7.81
Reward-Based	27.34	1.138	26.73	1.025	7.72

Reward-based triggering improves Pick-a-Pic IR from 1.062 to 1.138 over random triggering at similar NFEs, rather than merely increasing the number of searches. Always obtains higher values on these quality metrics but requires 16.72 NFEs; reward-based triggering uses 7.72, a reduction of 9.00, and is not a quality-loss-free replacement.

Key Findings

• Figure 3 on page 12 supports joint effects of depth and width, with deeper, narrower settings sometimes outperforming shallower, wider settings at similar NFEs; individual scatter values cannot be recovered reliably from the cache, so no numerical ablation values are invented.
• The reward-model ablation in Table 3 shows that AES guidance improves aesthetics without necessarily improving conditional alignment to the same extent; the scorer also changes trigger frequency and realized NFEs.
• In the COCO-2k check in Table 4 on page 14, SD2.1 Recall changes from DDIM's 55.85 to Ctrl-Z‡'s 55.20, while LPIPS changes from 75.00 to 75.54, using the original table's scales.
• These diversity results indicate no consistent degradation only within the evaluated sample size and budget range; they do not prove freedom from mode concentration under arbitrary rewards or larger search budgets.

Highlights & Insights

• Trigger timing is a meaningful sampler design choice: algorithms with similar candidate-generation procedures can differ substantially because they allocate computation to different steps.
• Depth and width are not interchangeable budget controls. More candidates increase local hit probability, whereas deeper rollback can change the structural region that candidates can reach.
• Including the ordinary forward step as a fallback avoids mandatory acceptance of random perturbations, but this local fallback is not a global guarantee about final quality or historical reward.

Limitations & Future Work

• The authors acknowledge that a fixed threshold can repeatedly penalize high-score plateaus; adapting the threshold to score scale or sampling stage is a direct improvement direction.
• Validation centers on two latent-space image-generation backbones; pixel-space EDM, video generation, and higher-budget scaling remain unvalidated extensions in this paper.
• Reader assessment: ImageReward is used for both search and part of evaluation, coupling IR gains to the objective; other scorers and compositional tests provide complementary evidence, not direct substitutes for human preferences.
• Reader assessment: the main tables do not provide a complete wall-clock breakdown, so lower NFEs cannot be converted directly into proportional reductions in user waiting time.
• The paper points to supplementary threshold and window ablations, but the supplied cache contains only the main paper and references; unavailable experimental details are not inferred here.

Related Work & Insights

• vs Resampling: it explores nearby states through shallow re-noising, while Ctrl-Z lets reward decide when to intervene and when to deepen the rollback; the contribution is not the first introduction of backward moves.
• vs Z-Sampling: it reinforces conditioning through unconditional inversion and conditional denoising; Ctrl-Z controls trajectories using random rollback, explicit scoring, and candidate retention.
• vs SOP: both can rank with ImageReward, but Ctrl-Z triggers from stagnation and adapts rollback depth; main-table advantages should be read alongside NFEs and metrics on which it does not win.
• Research direction: confidence in intermediate clean-image predictions could modulate thresholds to reduce rollbacks triggered by unreliable rewards; this is a reader proposal, not a component implemented in the paper.

Rating

• Novelty: 4/5. The main contribution combines reward-triggered search with adaptive rollback depth rather than introducing a new diffusion training paradigm.
• Experimental Thoroughness: 4/5. Two backbones, multiple benchmarks, and trigger and reward ablations are covered, but larger budgets, human evaluation, and complete latency analysis remain limited.
• Writing Quality: 3/5. The intuition and main tables are clear, but strict monotonicity claims require distinguishing the algorithm's fallback branch, and extracted equations need source verification.
• Value: 4/5. A tunable route to better text-to-image generation without changing model weights, with practical gains still dependent on reward quality and compute budget.