LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion¶

Conference: ECCV 2026
arXiv: 2603.14526
Code: https://zengqunzhao.github.io/LatSearch (Project Page)
Area: Video Generation / Diffusion Models / Inference-Time Scaling
Keywords: Video Diffusion, Inference-Time Scaling, Latent Reward Model, Reward-Guided Search, Golden Noise

TL;DR¶

LatSearch trains a latent reward model that can directly score the latent vectors of intermediate states in the denoising trajectory (rather than decoded videos), and utilizes it to drive "Reward-Guided Resampling and Pruning (RGRP)" in the latent space during inference. This achieves video generation quality comparable to or even better than the SOTA, while reducing runtime by up to 79%.

Background & Motivation¶

Background & Limitations of Prior Work: The success of "inference-time scaling" in LLMs has inspired the video diffusion community. In addition to simply increasing denoising steps, it has been discovered that "golden noise" exists—where certain initial noises naturally generate higher-quality videos. Consequently, a vast body of work invests extra compute during the inference stage to optimize or search for better initial noise. However, these methods fall into two categories, each with its own critical vulnerabilities. The first is noise optimization (e.g., FreeInit, FreqPrior), which injects noise, applies optical flow warping, or merges frequency domains of reference videos to add priors to initialization. However, once the denoising trajectory begins, they lack any mechanism to monitor and correct intermediate states, allowing errors introduced early on to accumulate along the long denoising trajectory. The second is noise search (e.g., VideoReward, EvoSearch), which generates multiple candidates and selects the best one using Best-of-N, beam search, or evolutionary algorithms. However, they evaluate candidates only on fully decoded videos. Repeatedly decoding full videos is computationally prohibitive, and the reward signals are delayed and sparse. Due to these high costs, stronger search algorithms are virtually unusable.

Key Challenge: Stronger search algorithms are precisely the key to unlocking controllability, sample efficiency, and generation quality. However, the prerequisite is that computational costs must be significantly reduced. The current paradigm, because it "can only evaluate the final video," pushes both costs and latency to their absolute limits. The root of the problem lies in the diffusion model's lack of ability to reliably evaluate intermediate latent states, which prevents the support of flexible strategies like early stopping. To make "stronger yet cheaper" search possible, this capability must first be established.

The Core Idea of this paper stems from this: instead of only evaluating at the end of the denoising trajectory, it is better to evaluate "partially denoised" latent vectors on the fly at any denoising step. To this end, the authors propose a latent reward model that scores intermediate latent states from three dimensions: visual quality (VQ), motion quality (MQ), and text alignment (TA), directly injecting process-level supervision into the denoising trajectory. This both prunes poor candidates as early as possible to reduce futile denoising steps and avoids the heavy cost of repeatedly decoding entire videos.

Method¶

Overall Architecture¶

LatSearch addresses the problem of "how to determine which candidate trajectory is more promising during denoising without decoding the video." The system is divided into two components: first, a latent reward model \(R_\psi\) is trained offline. It takes an intermediate latent tensor \(\bm{z}_t\), a timestep \(t\), and a text prompt \(p\) as inputs and outputs three scalar scores (VQ, MQ, and TA). During inference, this reward model drives the Reward-Guided Resampling and Pruning (RGRP) search. This search maintains \(N\) candidate trajectories in parallel, evaluates candidates at predetermined "scoring steps," resamples them based on reward-normalized probabilities (retaining unique seeds to save compute), and retains only the single optimal trajectory at the final scoring step based on cumulative rewards, which is then decoded into the final video.

A core challenge in training the latent reward model is that reward labels are inherently "video-level" (assigned to the decoded video), whereas the model must evaluate intermediate latent states that lack explicit semantics. This work uses similarity-grounded credit assignment to allocate video-level rewards to latent-level labels based on the cosine similarity between the intermediate states and the final clean latents, which are then used in joint training with regression and preference losses.

On the inference side, RGRP draws inspiration from importance sampling and Sequential Monte Carlo (SMC): multiple candidates are generated in the latent space, weighted by reward scores (acting as proxy importance weights), and resampled to balance exploration and exploitation, with final pruning extracting the "consistently high-reward" trajectory. The overall workflow is as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt + Base Noise"] --> B["Candidate Generation<br/>N Perturbed Trajectories Denoised in Parallel"]
    B --> C["Similarity-Grounded Latent Reward<br/>Scores Intermediate States on VQ/MQ/TA"]
    C -->|At Scoring Step S| D["Reward-Guided Resampling<br/>Softmax Weights + Uniqueness De-duplication"]
    D -->|Independent Denoising Between Steps| C
    C -->|At max(S)| E["Cumulative Weight Final Pruning<br/>Retain Only the Seed with Highest Cumulative Reward"]
    E --> F["Decode to Video"]

Key Designs¶

1. Similarity-Grounded Latent Reward Model: Enabling Video-Level Rewards to Supervise Non-Semantic Intermediate Latents

The pain point is straightforward: almost all reward scorers act on rendered videos and return video-level scores, but the proposed reward model must evaluate intermediate latents \(\bm{z}_t\) extracted along the denoising trajectory, where these intermediate states lack direct ground-truth labels. The proposed approach allocates rewards by "contribution": given a prompt \(p\) and the final clean latent \(\bm{z}_0\), the video-level reward vector \(\bm{r}=(r^{\mathrm{VQ}},r^{\mathrm{MQ}},r^{\mathrm{TA}})^\top=\mathcal{R}(\mathcal{D}(\bm{z}_0),p)\) is first obtained on the decoded video. Then, the closeness of the intermediate state to the final target is measured using a cosine similarity rescaled to \([0,1]\):

\[s_t=\frac{1}{2}\left(1+\frac{\langle\bm{z}_t,\bm{z}_0\rangle}{\//\bm{z}_t\//_2\,\//\bm{z}_0\//_2}\right)\in[0,1].\]

Then, each dimension of the video-level reward is discounted proportionally to \(s_t\) to construct the latent-level targets \(\tilde{\bm{r}}_t=s_t\cdot\bm{r}\), yielding the latent reward dataset \(\mathcal{D}_{\mathrm{latent}}=\{(\bm{z}_t,p,\tilde{\bm{r}}_t,t)\}\). The intuition is that the closer an intermediate state is to the final target (and the more it "contributes" to the final video), the higher the reward it should inherit. Structurally, the latent tensors are patchified into video tokens via lightweight 3D convolutions, the timestep \(t\) is mapped to a learnable embedding \(\mathbf{e}_t\) and concatenated into the sequence, the prompt is tokenized using instruction templates, and three special query tokens [VQ], [MQ], and [TA] are appended. The entire sequence is fed into a transformer backbone (implemented with Qwen2-VL-3B), and the hidden states of these three query tokens are passed through linear layers to obtain \(\hat{\bm{r}}=\mathrm{Linear}(h^{[\mathrm{VQ}]},h^{[\mathrm{MQ}]},h^{[\mathrm{TA}]})\). This allows evaluation to be completed inside the latent space, which is an order of magnitude cheaper than full video decoding (measuring only 0.84s per latent score in practice).

2. Regression Loss + Preference Loss: Anchoring Absolute Scores While Preserving Relative Rankings

Relying solely on regression presents an issue: it provides absolute supervision but does not guarantee the correct relative order among candidates—which is precisely what search algorithms rely on most. The regression loss is standard L2: \(\mathcal{L}^d_{\mathrm{reg}}=\|\hat{\bm{r}}^d-\tilde{\bm{r}}^d_t\|_2^2\). Based on this, inspired by RLHF, a preference loss is introduced: for each pair \((i,j)\) in a minibatch, let \(\Delta\hat{\bm{r}}^d_{ij}=\hat{\bm{r}}^d_i-\hat{\bm{r}}^d_j\), and the ground-truth preference is \(\bm{y}^d_{ij}=\mathbb{I}[\bm{r}^d_i>\bm{r}^d_j]\), then

\[\mathcal{L}^d_{\mathrm{pref}}=\frac{1}{|\mathcal{P}_d|}\sum_{(i,j)\in\mathcal{P}_d}\log\!\left(1+\exp\!\big(-(2\bm{y}^d_{ij}-1)\,\Delta\hat{\bm{r}}^d_{ij}\big)\right),\]

which essentially applies binary cross-entropy on pairwise score differences. The final loss is a weighted sum of three dimensions and two objectives: \(\mathcal{L}=\sum_{d}(\lambda^d_{\mathrm{reg}}\mathcal{L}^d_{\mathrm{reg}}+\lambda^d_{\mathrm{pref}}\mathcal{L}^d_{\mathrm{pref}})\) (where both weights are set to 1.0 in the implementation). Regression anchors the predictions to the absolute reward scale, while preference shapes the reward terrain to preserve relative orders. Ablation results show that adding the preference loss stably improves reward prediction accuracy by about 1–3% across different denoising steps, becoming more pronounced in later stages.

3. Reward-Guided Resampling + Uniqueness De-duplication: Believing in Rewards without Being Blinded by Them

With a functional scoring model, how is it used to select candidates? The most naive approach is to greedily select the highest-scoring candidate at each step, but since the reward model itself is imperfect, greedy selection over-relies on it. This work adopts an importance sampling approach: at the initial step \(T\), \(N\) candidate trajectories are first generated by adding isotropic perturbations to the base noise \(\bm{z}_T^{(0)}\): \(\bm{z}_T^{(i)}=\sqrt{1-\eta^2}\,\bm{z}_T^{(0)}+\eta\,\bm{\epsilon}_i\) (where \(\eta\) controls diversity). At a scoring step \(t\in\mathcal{S}\), after obtaining rewards \(r_i^{(t)}\), they are converted into softmax weights \(\pi_i^{(t)}=\exp(\tau r_i^{(t)})/\sum_k\exp(\tau r_k^{(t)})\) (where \(\tau\) is temperature), and then \(N\) samples are drawn with replacement from a multinomial distribution: \(\bm{n}^{(t)}\sim\mathrm{Multinomial}(N;\pi^{(t)})\). The key step is a uniqueness operator \(\mathrm{supp}(\cdot)\): it keeps only the distinct seeds sampled, denoted as \(\mathcal{I}^{(t)}=\{i\mid n_i^{(t)}>0\}\), discarding duplicate copies of the same seed—since duplicate trajectories would waste computation repeating identical denoising sequences. Under this, the survival probability of candidate \(i\) after de-duplication is \(1-(1-\pi_i^{(t)})^N\), which increases monotonically with its weight, naturally achieving a soft selection where "higher weights mean a higher chance of survival." Between scoring steps, the candidates independently perform several denoising updates.

4. Cumulative Weight Final Pruning: Determining Winners by "Consistent High Scores" Rather Than "Single-Step High Scores"

Looking only at the score of a single step can easily be misled by noise fluctuations. This work uses an additive cumulative criterion to aggregate evidence across scoring steps: \(c_i^{(t)}=c_i^{(t-1)}+\pi_i^{(t)}\) (with \(c_i^{(0)}=0\)). At the final scoring step \(t'=\max(\mathcal{S})\), if multiple candidates remain, the seed with the highest cumulative weight is selected: \(i^\star=\arg\max_{i\in\mathcal{I}^{(t')}}c_i^{(t')}\). Let \(\bm{z}_0=\bm{z}_0^{(i^\star)}\), and only this single trajectory is carried through the remaining denoising steps and decoded. This pruning step simultaneously improves quality and efficiency: it selects trajectories that are "consistently reliable" (rather than those with a fluke high score at a single step) and compresses the most time-consuming later denoising stages from \(N\) trajectories to 1, significantly saving compute. Ablations show that compared to a beam-search baseline relying on purely cumulative rewards, this probabilistic RGRP on average improves performance by 0.42%, illustrating that probabilistic selection balances exploration and exploitation better and avoids overfitting to cumulative rewards.

Loss & Training¶

The latent reward model is initialized with Qwen2-VL-3B, with a learning rate of 1e-4, decreasing to 1e-5 at the 10th epoch, a batch size of 4, and completion at the 15th epoch. Regression and preference losses are equally weighted (coefficients are both 1.0). Data construction: 1000 prompts that do not overlap with VBench-2.0 are sampled, and 5000 videos are generated using different random seeds. Meanwhile, the latents at selected timesteps and their similarities to the final latent are saved, divided 8:2 into training/testing. The video generation backbone is Wan2.1-1.3B (50 denoising steps, CFG=5.0, 33 frames, 832×480). The default scoring schedule is at timesteps {10,15,20}, and experiments are run on A100s.

Key Experimental Results¶

Main Results¶

We compare with inference-time scaling methods on VBench-2.0 (five dimensions: Creativity, Commonsense, Controllability, Human Fidelity, Physics). † denotes the use of DPM-Solver++. ⚠️ Note that the arrow annotations in the original Table 1's caption are suspected to be reversed (marking ↑ as "performance degradation" and ↓ as "performance improvement"); the table below follows the actual numerical values, with the relative change to the baseline average score in parentheses.

Method	Avg Score	Inference Time (s)	Description
Baseline (Wan2.1-1.3B)	51.90	77.21	Single-trajectory baseline
FreeInit [ECCV'24]	49.82 (−2.08)	308.87 (×4.00)	Noise optimization, drops performance and 4× slower
FreqPrior [ICLR'25]	50.37 (−1.53)	142.46 (×1.85)	Noise optimization, still drops performance
VideoReward [NeurIPS'25]	52.80 (+0.90)	283.63 (×3.67)	Best-of-N search
EvoSearch† [arXiv'25]	55.01 (+3.11)	783.76 (×10.15)	Evolutionary search, high quality but 10× slower
LatSearch (Ours)	53.84 (+1.94)	182.43 (×2.36)	Ours
LatSearch† (Ours)	55.25 (+3.35)	164.41 (×2.13)	Ours best configuration

Key conclusions: Noise optimization methods do not significantly increase computation but lack an effective validator. Simply feeding the temporal features of latents back to the initial noise drops performance instead (FreeInit −2.08, FreqPrior −1.53). Search-based methods improve quality but at an exorbitant cost (EvoSearch is 10× slower). LatSearch† achieves the highest score of 55.25 (+3.35) using ×2.13 compute. Under equivalent compute, it outperforms FreqPrior by 2.44%, and compared to EvoSearch, it lags by only 0.24% in quality while being 4.77× faster. The runtime breakdown (Table 7) highlights the source of superiority: LatSearch's decoding time is almost identical to the baseline (1.88s vs 1.85s), adding only a lightweight latent reward evaluation (1.03s), whereas EvoSearch requires repeated full decoding (DiT 756.66s + Decoding 31.99s, totaling 790.57s).

Ablation Study¶

The core ablation evaluates the individual contributions of the "Preference Loss (PL)" and the "RGRP" (Table 4):

Configuration	PL	RGRP	Avg Score	Description
Baseline	✗	✗	51.90	Single-trajectory
+ PL only	✓	✗	52.35 (+0.45)	Beam-search style, only adding preference loss reward model
+ RGRP only	✗	✓	53.42 (+1.52)	Regression reward model + RGRP search
Full	✓	✓	53.84 (+1.94)	Full model

Adding RGRP (with regression reward model) to the baseline yields +1.52, indicating that the search mechanism itself contributes the most. Further adding the preference loss to strengthen the reward model gains an extra +0.42 to 53.84, validating that "a stronger reward model benefits the search process more." RGRP outperforms the beam-search style (which deterministically retains fixed seeds based only on cumulative rewards) by an average of 0.42%, with only a minimal increase in compute.

Another ablation compares credit assignment strategies (Table 5, how to allocate video-level rewards to intermediate states):

Strategy	Avg Score	Description
Baseline	51.90	—
Uniform	52.31 (+0.41)	Equal weight across steps, almost ineffective
Exponential	52.11 (+0.21)	Exponential weighting, early bias
L2 Error	52.74 (+0.84)	L2 error weighting
Cosine Similarity	53.84 (+1.94)	Ours, best

Key Findings¶

The search mechanism is the main driver; the reward model is the gain: RGRP alone contributes +1.52 (accounting for the vast majority of the +1.94 in the full model), and the preference loss adds another +0.42. The two are orthogonally stackable.
Cosine similarity credit assignment significantly outperforms naive schemes: Naively spreading rewards uniformly across all timesteps (Uniform +0.41 / Exponential +0.21) is virtually ineffective, whereas cosine similarity (+1.94) is far superior because it aligns better with semantic consistency. The Appendix further shows that back-projecting the predicted latent rewards to video-level rewards yields an approximation error that decreases monotonically along the denoising trajectory—later latent states are better predictors of the final quality, which explains why the scoring schedule avoids extremely early steps.
Search budget N shows returns but saturates: N=4 \(\rightarrow\) 6 brings obvious improvements (52.81 \(\rightarrow\) 53.84), while 6 \(\rightarrow\) 8 shows diminishing returns (53.84 \(\rightarrow\) 54.13). Since evaluation takes place in the latent space and avoids duplicate decoding, the computational growth of increasing N is moderate.
Scoring schedule is the sweet spot: {10,15} lacks sufficient coverage (+0.27), {10,15,20,25,30} degrades instead (+0.44) due to the accumulation of uncertainty in similarity targets, and {10,15,20}, which is well-spaced in the middle range, yields the best result (+1.94). The temperature \(\tau \in \{0.5, 1.0, 2.0\}\) has very little effect on the results, indicating that the resampling is robust to temperature.
Generalization across backbones: Because it operates solely in the latent space and does not depend on architecture-specific components, LatSearch consistently improves Wan2.1-14B (52.58 \(\rightarrow\) 53.61), CogVideoX-2B (45.13 \(\rightarrow\) 48.14, +3.01), and CogVideoX-5B (48.13 \(\rightarrow\) 50.21, +2.08).
Consistent with human preferences: In a user study with 40 video pairs and 30 evaluators, LatSearch was preferred in visual quality/motion quality/text alignment by 72.15%/75.29%/78.51% (averaging 75.32%), aligning with automatic metrics.
Extrapolation in duration/resolution: Gains of +1.94/+1.21/+0.87/+1.37 are observed for 2s/3s/4s/5s, and +1.94 and +0.80 for 480p and 720p, respectively. Gains expectedly decrease with longer durations and higher resolutions.

Highlights & Insights¶

Moving evaluation to the latent space is the key efficiency lever: Previously, the main bottleneck of search-based methods was repeated full-video decoding (EvoSearch's decoding takes 31.99s). LatSearch moves evaluation to the latent state, taking only 0.84s per evaluation, and performs decoding only once for the final winner. This is the fundamental reason it is 4.77× faster while matching quality, a concept transferable to any generative search where candidate evaluation is costly.
Similarity grounding is a clever way of "generating labels out of nothing": Intermediate latent states lack ground-truth rewards. Using cosine similarity to discount video-level rewards according to "contributions to the final destination" makes a seemingly impossible-to-supervise problem trainable, and this approximation error is empirically proven to monotonically converge along the denoising trajectory rather than being purely arbitrary.
Engineering awareness of not over-relying on the reward model: Knowing that the reward model is imperfect, the authors avoid greedy selection and instead employ softmax probabilistic resampling + uniqueness de-duplication + cumulative weight pruning. This beautifully harmonizes "trusting the rewards" with "preserving exploration diversity." This SMC-flavored soft selection is more stable than the deterministic preservation of beam search.
Plug-and-play: It only needs to evaluate intermediate denoising states without touching the generative backbone. It is naturally compatible across multiple families, such as Wan and CogVideoX, lowering usability barriers.

Limitations & Future Work¶

No theoretical optimality guarantee: RGRP draws on SMC, but because the latent reward model is a learned approximation, establishing convergence guarantees is difficult, and the authors explicitly do not claim formal optimality.
Credit assignment is still an approximation: Cosine similarity performs best in ablations but is merely a proxy for "true semantic contribution." The authors suggest replacing it in the future with a lightweight, contrastive/self-supervised temporal similarity estimator to directly overcome this approximation bottleneck.
Gains decay with difficulty Improvements decrease under longer durations and higher resolutions (only +0.80 for 720p), indicating diminishing marginal returns under more challenging settings.
Limited improvement in controllability: The improvement in Controllability is generally very small across tables (e.g., only +0.70 for full), with the main gains concentrated in creativity, commonsense, and physics.
Extensible directions: The visual quality and motion quality dimensions of the reward model are modal-agnostic. The authors suggest that by replacing the text alignment target with task-specific alignment modules, the method can be extended to audio-video generation, video editing, and instruction-guided transformation.

vs FreeInit / FreqPrior (Noise Optimization): These only apply frequency/temporal priors to the initial noise at the beginning of the trajectory and cannot correct errors once denoising starts. This work continuously evaluates and resamples mid-trajectory, successfully improving quality without dropping performance (in contrast, the former two actually drop performance by −2.08 / −1.53).
vs VideoReward (Best-of-N Search): It must denoise every candidate to the end and decode them to videos, using video-level rewards to pick the best, which introduces huge decoding overhead (×3.67 compute). This work scores directly on latents and only decodes the winner, producing higher quality under the same budget.
vs EvoSearch (Evolutionary Search): Evolutionary search offers the highest quality but is 10× slower. This work lags by only 0.24% in quality while being 4.77× faster, truly lowering the compute barrier for "strong search".
vs Latent Reward Works Like DOLLAR: Prior work also used latent reward models for few-shot video generation, but only evaluated the final denoised latent. This work evaluates intermediate states across timesteps, achieving finer-grained process-level guidance. ⚠️ The name of this baseline is kept as-is from the original citation.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of "latent space intermediate state scoring + probabilistic resampling scaling" is a clear and practical new idea in video inference-time scaling, though similarity grounding and SMC-style search both have predecessors.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ VBench-2.0 5-dimensional main comparison, runtime breakdown, cross-backbone testing on three families, duration/resolution extrapolation, multiple ablation groups for credit assignment/temperature/scheduling, plus user studies, covering everything comprehensively.
Writing Quality: ⭐⭐⭐⭐ The motivation and method are clearly explained with standardized formulas. Minor issues include the reversed arrow labels in the main results table caption and slight inconsistencies in notation (RGRP/PGRP), which can lead to misinterpretation.
Value: ⭐⭐⭐⭐ "Nearly 5× faster for similar quality" possesses clear practical value for video diffusion inference deployment, being plug-and-play and applicable across backbones.