Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/rogerioagjr/psp
Area: Image Generation
Keywords: Diffusion Models, Flow Matching, Inference-Time Scaling, Seed Pruning, Reward-Guided Generation

TL;DR¶

Progressive Seed Pruning (PSP) front-loads compute to explore an expanded pool of initial noise seeds, progressively eliminating unpromising candidates via zero-overhead intermediate denoised estimates and black-box rewards to achieve state-of-the-art prompt alignment under a matched compute budget.

Background & Motivation¶

Conditional visual generation has increasingly centered on diffusion and flow-matching backbones. Drawing inspiration from large language models, where test-time search, sampling, and verification deliver substantial inference-time scaling gains, generative vision models also seek principled strategies for allocating extra inference compute. However, the conventional lever of simply increasing denoising steps yields diminishing returns; extensive empirical evidence demonstrates that the initial random noise seed exerts an outsized influence on macro-composition and high-level semantic alignment. Consequently, steering seed selection using black-box reward models has emerged as the primary vehicle for test-time scaling.

Existing test-time scaling paradigms suffer from structural and mechanistic limitations. Standard Best-of-N (BoN) fully denoises all candidate seeds, expending immense compute on trajectories destined to lose. Meanwhile, recent gradient-free alternatives based on importance sampling (e.g., FK-Steering) and tree search (e.g., DSearch) strictly enforce a constant memory constraint by propagating a fixed number of active particles across all denoising steps. To generate diverse candidate branches, these methods rely on injecting stochastic perturbations, which severely degrades sample quality in modern flow-matching models (such as Stable Diffusion 3.5) governed by deterministic rectified-flow ODE solvers. Crucially, because coarse layout and prompt-alignment constraints are established early in the trajectory, constant-memory allocation artificially curtails exploration precisely when seed diversity matters most.

This paper addresses this bottleneck by relaxing the constant-memory restriction, observing that modern multi-GPU server infrastructure naturally accommodates elastic batching. Furthermore, standard diffusion and flow solvers inherently produce single-step clean estimates without extra model forward passes, enabling cheap intermediate reward evaluations. Core idea: Progressive Seed Pruning (PSP) front-loads inference compute by initiating an oversized pool of noise seeds, scores intermediate clean estimates via black-box rewards on a predetermined schedule, aggressively prunes unpromising candidates, and dedicates remaining compute to fully denoise the survivors under a matched total budget.

Method¶

Overall Architecture¶

PSP reallocates inference compute toward early seed exploration while preserving the total number of generator updates, quantified by the effective compute multiplier $\bar{N} = C/T$. The complete pipeline operates through four orderly stages: initial parallel propagation across an expanded candidate seed pool, zero-overhead clean-image extrapolation from generator outputs, Top-K survivor pruning guided by a black-box reward model, and deterministic final denoising of the retained candidates. By utilizing fixed pruning milestones and survivor counts, PSP maintains predictable memory and latency profiles suitable for elastic multi-GPU deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input prompt c & enlarged seed pool<br/>Sample k_T initial Gaussian seeds"] --> B["Front-loaded exploration<br/>Advance concurrent partial trajectories"]
    B --> C["Zero-overhead clean extrapolation<br/>Derive x_0 estimates from solver states"]
    C --> D["Black-box scoring & schedule pruning<br/>Evaluate rewards & filter via Top-K"]
    D --> E["Survivor denoising continuation<br/>Focus remaining steps on top candidates"]
    E --> F["Output highest-reward generation"]

Key Designs¶

1. Front-loaded compute allocation: Trading variable particle count for doubled seed exploration Prior importance-sampling and tree-search methods enforce a rigid constant particle count throughout inference, severely restricting the initial candidate pool. PSP targets the empirical reality that coarse semantic features and prompt alignment are dictated in the earliest phases of reverse sampling. By relaxing constant-memory constraints and initializing $k_T = 2\bar{N}$ seeds, PSP doubles the candidate exploration space relative to BoN at the identical aggregate compute budget. This broad exploration substantially minimizes the risk of getting trapped in suboptimal mode basins caused by poor initial noise realizations.

2. Zero-overhead intermediate clean extrapolation: Repurposing solver outputs as evaluation proxies Evaluating candidate viability at intermediate noisy steps without executing complete rollouts is challenging. PSP leverages the mathematical formulation of diffusion and flow solvers. In diffusion models, the noise prediction directly yields the clean expectation $\hat{x}_0(x_t, t, c) = \mathbb{E}[x_0 \mid x_t, t, c]$; in flow matching, the velocity field $v_\theta(x_t, t, c)$ affords a one-step linear extrapolation to the zero-noise endpoint: $$\hat{x}_0(x_t, t, c) = x_t - \epsilon_t v_\theta(x_t, t, c)$$ Because $\hat{x}_0$ is already computed internally during the standard solver step, generating intermediate proxy images incurs zero additional generator evaluations. The decoded proxies are directly scored by the black-box reward model: $$s_t = r(\hat{x}_0(x_t, t, c), c)$$

3. Predetermined geometric pruning schedule: Aligning ranking stability with deployment predictability Continuous online particle adaptation introduces severe scheduling overheads in production serving. PSP adopts a predetermined geometric halving schedule inspired by successive halving in multi-armed bandits. For the default compute multiplier $\bar{N} = 4$, the procedure instantiates $8$ initial seeds, prunes the pool to $4$ survivors at $25\%$ progress, prunes again to $2$ survivors at $50\%$ progress, and denoises those $2$ final trajectories to completion. Predetermined checkpoints and batch sizes ensure deterministic VRAM footprints and execution times per interval, making PSP exceptionally friendly to production inference engines.

4. Fully deterministic trajectory solver: Eliminating stochastic degradation and enabling offline schedule simulation Unlike particle resampling techniques that require stochastic samplers to branch into distinct child states, PSP operates with purely deterministic solvers (DDIM with $\eta = 0$ for diffusion models; Euler Discrete for flow matching). This eliminates the trajectory distortion that stochastic noise inflicts on rectified-flow models. Furthermore, determinism guarantees that intermediate trajectories and rewards are invariant for a given seed pool. Consequently, practitioners can precompute trajectories once on a validation set and perform exhaustive offline grid searches over pruning schedules in seconds without reinvoking the underlying generator.

Loss & Training¶

PSP is strictly training-free and gradient-free; it does not update or fine-tune backbone weights. Black-box guidance is provided by ImageReward, evaluated exclusively at the designated pruning intervals. Peak VRAM overhead during intermediate decoding is effectively managed using standard VAE slicing (sequential decoding across the batch), adding less than $0.15\text{ s}$ of latency while maintaining memory footprints comparable to standard BoN.

Key Experimental Results¶

Main Results¶

On the GenEval benchmark, under a matched compute multiplier ($\bar{N}=4$), PSP was evaluated against standard inference ($\bar{N}=1$), BoN, importance sampling (FK-Steering), and tree search (DSearch, BFS) across Stable Diffusion v1.5, SDXL, and Stable Diffusion 3.5. Metrics include ImageReward guidance score, HPSv2, automated GenEval prompt alignment, and human evaluation via crowdsourced majority voting across 24,885 ratings.

Model Backbone	Sampler / Method	Steps $T$	Multiplier $\bar{N}$	ImageReward $\uparrow$	HPSv2 $\uparrow$	GenEval $\uparrow$	Human Eval $\uparrow$
SD v1.5	Standard	64	1	-0.159	0.257	0.434	0.431
SD v1.5	Best-of-N	64	4	0.655	0.273	0.542	0.594
SD v1.5	FK-Steering	64	4	0.640	0.261	0.531	0.569
SD v1.5	DSearch	64	4	0.783	0.275	0.506	0.514
SD v1.5	BFS	64	4	0.820	0.263	0.564	-
SD v1.5	PSP (Ours)	64	4	0.827	0.278	0.574	0.624
SDXL	Standard	64	1	0.431	0.275	0.529	0.531
SDXL	Best-of-N	64	4	1.098	0.290	0.629	0.682
SDXL	FK-Steering	64	4	1.189	0.284	0.627	0.676
SDXL	DSearch	64	4	1.186	0.301	0.589	0.649
SDXL	BFS	64	4	1.247	0.285	0.636	-
SDXL	PSP (Ours)	64	4	1.224	0.294	0.645	0.713
SD 3.5	Standard	32	1	1.045	0.297	0.713	0.787
SD 3.5	Best-of-N	32	4	1.336	0.304	0.747	0.831
SD 3.5	FK-Steering	32	4	1.294	0.284	0.742	0.837
SD 3.5	DSearch	32	4	1.144	0.297	0.704	0.824
SD 3.5	BFS	32	4	1.343	0.285	0.747	-
SD 3.5	PSP (Ours)	32	4	1.380	0.306	0.747	0.841

Ablation Study¶

Benchmarking on a single NVIDIA H200 GPU quantifies computational runtime and VRAM overhead, alongside interaction experiments with Direct Preference Optimization (DPO) fine-tuned models.

1. Latency and Peak VRAM Overhead (NVIDIA H200)

Model Backbone	Method Config	Multiplier $\bar{N}$	Runtime (s)	Peak VRAM (GiB)	Note
SD v1.5	BoN	4	2.65	4.29	Matched compute baseline
SD v1.5	PSP	4	2.99	5.30	Explores 8 seeds; +12.8% runtime overhead
SD v1.5	BoN	8	4.70	5.31	Full exploration baseline; PSP is 1.57× faster
SDXL	BoN	4	12.48	8.28	Matched compute baseline
SDXL	PSP	4	13.74	13.27	Explores 8 seeds; +10.1% runtime overhead
SDXL	BoN	8	23.90	10.79	PSP delivers ~1.74× speedup over full exploration
SD 3.5	BoN	4	35.97	29.85	Matched compute baseline
SD 3.5	PSP	4	37.59	34.85	Explores 8 seeds; +4.5% runtime overhead
SD 3.5	BoN	8	71.71	33.86	PSP approaches 2× speedup with superior alignment

2. Complementary Gains with DPO Fine-Tuning

Base Model	DPO Fine-Tuned	Sampler Strategy	Steps $T$	Multiplier $\bar{N}$	ImageReward $\uparrow$	GenEval $\uparrow$
SD v1.5	No	Standard	64	1	-0.159	0.434
SD v1.5	Yes	Standard	64	1	-0.022	0.454
SD v1.5	No	PSP	64	4	0.827	0.574
SD v1.5	Yes	BoN	64	4	0.751	0.562
SD v1.5	Yes	PSP	64	4	0.907	0.593
SDXL	No	Standard	64	1	0.431	0.529
SDXL	Yes	Standard	64	1	0.894	0.581
SDXL	No	PSP	64	4	1.224	0.645
SDXL	Yes	BoN	64	4	1.300	0.657
SDXL	Yes	PSP	64	4	1.365	0.667

Key Findings¶

Broad initial exploration outperforms intermediate resampling: Across all backbones, PSP consistently leads in GenEval prompt alignment and human preference. While BFS and DSearch occasionally drive up raw reward scores, they suffer from reward hacking caused by local stochastic branching. PSP avoids perturbing the trajectory and focuses purely on selecting superior global starting points.
Diminishing regret under scaling: As $\bar{N}$ scales from $1$ to $16$, PSP displays sustained performance scaling. The regret relative to the theoretical upper bound (the best fully denoised sample among all $2\bar{N}$ seeds) drops steadily as budget increases, confirming that early intermediate scoring is robust enough to prune safely.
Orthogonality to training-time alignment: Applying PSP ($\bar{N}=4$) on an un-finetuned model exceeds standard sampling on a DPO-trained backbone. Furthermore, layering PSP on top of DPO yields substantial additive gains, demonstrating that inference-time search complements parameter fine-tuning.
Applicability to discrete prompt rewrites: Replacing initial candidate seeds with diversified prompt rewrites generated by an LLM (ChatGPT) allows PSP to outperform BoN under matched compute, proving that the progressive pruning principle extends to general discrete initial conditions.

Highlights & Insights¶

Identifying the "constant-memory" restriction as an artificial bottleneck for generative models, showing that front-loading compute into early seed exploration dramatically increases search efficacy.
Exploiting single-step denoising extrapolations $\hat{x}_0$ allows zero-overhead intermediate scoring directly from the generator's internal forward pass.
Full determinism avoids stochastic trajectory degradation on modern flow-matching architectures and unlocks instant offline schedule simulation from cached features.

Limitations & Future Work¶

Single scalar reward dependency: PSP assumes a single scalar objective, making multi-attribute or spatial-geometric constraints (e.g., ControlNet-level pixel-precise edge alignment) harder to optimize directly without composite score formulations.
Diminishing returns on late-stage fine detail: Coarse compositional attributes are established early, where seed pruning excels; fine-grained texture and aesthetic refinements crystalize late in the trajectory, where early pruning provides narrower advantages.
Future directions: Integrating multi-reward Pareto-optimal filtering and exploring adaptive, uncertainty-aware pruning steps.

vs Best-of-N (BoN): BoN fully denoises all candidates; PSP progressively prunes unpromising seeds using geometric halving, exploring twice the seed pool under identical compute.
vs FK-Steering (Feynman-Kac Steering): FK-Steering maintains constant particle parallelism and depends on stochastic resampling, which harms rectified-flow ODE solvers; PSP enforces determinism and reallocates compute temporally.
vs DSearch / BFS (Tree Search): Tree search expands dynamic branches mid-trajectory, complicating batch scheduling and risking reward gaming; PSP relies on static pruning schedules with highly predictable serving characteristics.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Challenging the constant-memory dogma with an intuitive front-loaded seed pruning paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous coverage of diffusion and flow models with automated metrics, latency profiling, and 24k+ human ratings.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically concise, and thorough in addressing practical systems trade-offs.
Value: ⭐⭐⭐⭐⭐ Training-free, gradient-free, deterministic, and readily deployable in real-world image generation pipelines.