Early Estimation of Language to Latent Alignment in Diffusion Models¶

Conference: ECCV 2026
Paper: ECCV paper page
Project: NoisyCLIP
Area: Image Generation
Keywords: diffusion models, language-to-latent alignment, noise-aware contrastive learning, early stopping, Best-of-N

TL;DR¶

NoisyCLIP fine-tunes the visual encoder of a dual-encoder model on moderately noisy diffusion states to assess candidate-prompt alignment early, reducing the aggregate Best-of-6 denoising budget from 300 to 150 steps while limiting the VQAScore loss against the full-generation CLIP baseline to approximately 2%, as reported in the paper.

Background & Motivation¶

Text-to-image diffusion models can produce images with different levels of semantic alignment from different random seeds, even under the same prompt. Best-of-N therefore offers a straightforward form of test-time scaling: generate several candidates and select the best with CLIP or another evaluator. Conventional selection, however, happens after every image is complete. Candidates that are ultimately discarded still consume an entire denoising trajectory, and this waste increases with the number of candidates.

Early selection is not simply a matter of moving the final CLIP call to an intermediate step. Early diffusion latents are dominated by noise, and ordinary vision-language encoders may not recover stable semantics even after the latents are converted to RGB. Their scores consequently fail to separate promising trajectories from poor ones. Training only on nearly finished states is easier but does not address early recognition; training only on highly noisy states can compromise later assessment. The central problem is adapting the scorer to noise stages, not retraining the generator.

The authors therefore save real generation trajectories, train the visual branch on noisy states paired with their prompts, and insert the resulting scorer into candidate selection. Core Idea: learn text alignment from moderately noisy latent representations so that poorly aligned trajectories can be discarded early and subsequent denoising compute can be concentrated on the most promising candidate.

Method¶

Overall Architecture¶

NoisyCLIP is an alignment scorer external to the generator. It takes a prompt and an intermediate latent from a particular denoising stage and returns their similarity. During training, the authors generate and save SDXL trajectories, fine-tune the visual encoder on a selected noise range, and freeze the text encoder. At inference time, multiple candidates are generated partway before the scorer selects a single candidate to finish.

Intermediate latents cannot be fed directly into the original image encoder, so a representation conversion is required. SDXL's four-channel latents are mapped to RGB through a linear transformation. For transfer to SD3.5 and FLUX.1, the corresponding model's VAE decodes the latents instead. Thus, language-to-latent alignment refers to evaluating the semantics carried by latent states; it does not mean that every experiment bypasses decoding and directly processes raw latents.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Prompts["Training prompts"] --> Sampling["Mid-range noise sampling"]
    Sampling -->|Training samples| Scoring["Noise-aware dual-encoder scoring"]
    Text["Frozen text encoder"] --> Scoring
    Candidates["Intermediate latents of<br/>inference candidates"] -->|Score after RGB conversion| Scoring
    Scoring -->|Rank at inference time| Selection["Single-candidate continuation"]
    Selection --> Output["Complete the highest-scoring candidate<br/>Stop the other trajectories"]

Training samples in the diagram are used to learn the scorer; they do not have to be rebuilt during inference. Selection also requires neither the final image nor VQAScore as an input. VQAScore is used only in experimental evaluation to measure how well the selected result matches the prompt, rather than treating the selection similarity itself as the final performance metric.

Key Designs¶

1. Mid-range noise sampling: learn from real trajectory states that remain ambiguous but already contain semantics

The authors use rewritten, denser CC12M captions as SDXL prompts and save each prompt, final image, and full sequence of intermediate latents. These are states actually traversed by the generator, rather than ordinary images with noise added on demand. The objective is not image reconstruction: it is to preserve the correspondence between a prompt and its visual representation despite noise.

The default training interval is 20–29. Figure 5 shows that training on interval 10–19 favors early states but deteriorates substantially in the final two inference intervals. Interval 40–49 benefits late states and final images, whereas 20–29 supports both earlier assessment and stable later performance. These interval labels follow the paper's description of denoising progress; they should not be interpreted as universal noise levels across schedulers or confused with a diffusion time variable that decreases during sampling.

2. Noise-aware dual-encoder scoring: retain the text space while adapting the visual branch to noisy inputs

The text branch retains its pretrained parameters, while the visual branch is initialized from CLIP and fine-tuned on RGB representations converted from noisy latents. Each noisy representation and its own prompt form a positive pair; other prompts within the batch supply negatives. InfoNCE encourages the model to recover the correct cross-modal correspondence despite noise. Freezing the text branch concentrates optimization on the unfamiliar visual distribution. The authors also note that the text branch can be shared with the diffusion model when the encoders are compatible; this is not an automatic property of every generator architecture.

The score remains cosine similarity. The following normalized notation restates the definition in Section 3.1, with \(\Phi\) denoting latent conversion, \(\nu\) the visual encoder, and \(\tau\) the text encoder. This is neither an additional regression head nor a model trained directly against final VQAScore labels.

\[ S(z_t,y)=\frac{\nu(\Phi(z_t))^{\mathsf T}\tau(y)}{\|\nu(\Phi(z_t))\|_2\,\|\tau(y)\|_2}. \]

Accordingly, predicting final alignment means that intermediate scores can distinguish candidates whose final images will be better aligned. It does not mean that the score has been proven equal to final quality or calibrated as a success probability. The paper also transfers noise adaptation to SigLIP, producing NoisySigLIP; its image-selection ability and fine-grained text discrimination must be evaluated separately.

3. Single-candidate continuation: select at a shared checkpoint and terminate the other trajectories

Several random seeds are started for the same prompt. After each candidate reaches a predefined stage, NoisyCLIP ranks them and only the highest-scoring candidate completes the remaining steps. Early stopping applies to rejected trajectories, while the winning image still undergoes the full denoising process. The method neither returns an unfinished image nor updates generator parameters at every step. The main evidence concerns this single intermediate selection, not a multiround adaptive search procedure.

To make the source of compute savings explicit, let \(N\) be the number of candidates, \(T\) the length of a full trajectory, and \(k\) the number of completed steps at selection. The counting rule in Section 5.3 can be summarized as:

\[ C_{\mathrm{early}}=Nk+(T-k)=T+(N-1)k,\qquad C_{\mathrm{full}}=NT. \]

This expression summarizes the paper's accounting rather than including the time spent on scoring, representation conversion, VAE decoding, or parallel scheduling. Earlier selection saves more steps but has less semantic information available. NoisyCLIP supplies a more reliable ranking than frozen CLIP at that decision point; it does not reduce the cost of an individual diffusion step.

A Worked Example¶

Suppose two SDXL candidates are started for the same prompt, with 50 steps required for each complete trajectory. After both reach step 25, they have consumed 50 aggregate steps. The scorer compares their intermediate representations with the prompt and retains only the higher-scoring candidate. Another 25 steps complete the winner, giving a total cost of 75 rather than the 100 required by full Best-of-2, a saving of 25%. This is the cost example explicitly given in the paper, not an additional measured quality result.

The same accounting explains Best-of-6: selection after 20 completed steps costs \(6\times20+30=150\), whereas completing all candidates costs \(6\times50=300\). This calculation explains the half-budget setting but does not guarantee that every prompt retains the candidate with the best final image. That claim must be tested through VQAScore evaluation.

Loss & Training¶

The paper specifies InfoNCE with matching prompts and noisy representations as positives and mismatched within-batch combinations as negatives, updating only the visual encoder. The cache does not provide fully verifiable temperature settings or bidirectional-loss implementation details, so no additional, apparently exact training formula is supplied here.

Training uses 50,000 latent samples, 10 epochs, a batch size of 128, and a learning rate of \(5.7\times10^{-5}\). It applies cosine annealing, a 0.1 warm-up ratio, and 0.1 weight decay on an NVIDIA A100 40G. The implementation section describes uniform sampling from generated images, while the method section separately specifies the default training interval of 20–29; this does not establish uniform training over all timesteps.

The evaluation data also target semantic discrimination during generation. Noisy-Conceptual-Captions contains 1,000 prompts, with 4 generated images and 4 non-factual distractor prompts per original prompt. Retrieval selects among 1 correct and 4 incorrect descriptions. Noisy-GenAI-Bench contains 1,600 prompts, comprising 730 basic and 870 advanced prompts, each expanded into 10 generation trajectories. The main BoN analysis in the body uses the basic prompts.

Key Experimental Results¶

Main Results¶

The following results are from Table 1. R@1 is the proportion of examples for which the correct prompt ranks first among five descriptions. BoN VQAScore measures the selected image's prompt alignment; higher is better for both metrics. The intermediate-stage results at 21–30 in Table 1 must not be conflated with the basic-prompt cost-curve scores in Figure 6.

Scorer and evaluation stage	Factual consistency R@1	BoN VQAScore
CLIP L14, intermediate stages 21–30	0.145	0.657
NoisyCLIP, intermediate stages 21–30	0.506	0.709
SigLIP L16, intermediate stages 21–30	0.147	0.652
NoisySigLIP, intermediate stages 21–30	0.194	0.718
CLIP L14, final image at step 50	Not reported	0.747
SigLIP L16, final image at step 50	Not reported	0.764

NoisyCLIP improves intermediate R@1 over CLIP by 0.361, or 36.1 percentage points, reaching approximately 3.49 times the baseline. BoN VQAScore increases from 0.657 to 0.709, an absolute gain of 0.052. The intermediate score of 0.709 is approximately 94.9% of final-image CLIP's 0.747. This is a ratio of scores across evaluation stages, not a claim that 94.9% of candidate selections are correct.

Ablation Study¶

The following table summarizes the training-range analysis in Section 5.2 and Figure 5. The cached body does not retain the heatmap's individual cell values, so the table reports only verifiable intervals, explicit numbers, and author-described trends rather than presenting trends as exact ablation scores.

Training configuration	Evaluation range or value	Conclusion supported by the paper
Frozen CLIP, no noisy training	Best intermediate result over 1–49: 0.696; final image: 0.747	Clean-image capability does not transfer directly to intermediate states
NoisyCLIP, interval 10–19	Early inference ranges; the text reports gains of up to 4 points	Better early assessment but substantial deterioration in the final two ranges
NoisyCLIP, interval 20–29	Default middle training interval	Stronger earlier performance and late results close to low-noise training
NoisyCLIP, interval 40–49	Late ranges and final images	Best overall in these stages; readable cell-level values are unavailable

Figure 6 separately reports the Best-of-6 cost trade-off: full-generation CLIP scores 0.85 at 300 steps, while NoisyCLIP scores 0.86 at the same budget. At 150 steps, the authors report an approximately 2% VQAScore loss against full-generation CLIP and LatentCLIP-8. The body does not state the exact score at that point; reverse-calculating it from “2%” would not produce an independently reported measurement.

Key Findings¶

The training noise range matters: the earliest and latest ranges introduce stage preferences. Middle-range training is useful for covering a broader portion of inference, not because it is optimal in every interval.
Figure 3 describes clear separation between better-aligned and poorer candidates around latent stage 20 with NoisyCLIP, compared with approximately stage 40 for frozen CLIP.
The tasks are not interchangeable: NoisySigLIP's BoN score of 0.718 exceeds NoisyCLIP's 0.709, but its R@1 is only 0.194 versus 0.506.
For zero-shot transfer to SD3.5 and FLUX.1, the body and Figure 9 report alignment gains of +2% and +1.8%, respectively, alongside a 57% reduction in denoising steps. These are still not end-to-end latency measurements.

Highlights & Insights¶

Train the evaluator on the distribution where decisions are made. Ordinary CLIP's effectiveness on final images does not guarantee useful rankings of noisy states; adapting the visual branch with a modest training set directly addresses this mismatch.
Connect quality prediction to an explicit compute decision. Intermediate alignment is not merely a monitoring curve: it determines which trajectories deserve further computation, making its utility measurable through the relationship between quality and aggregate denoising budget.
Evaluate visual ranking and factual text discrimination separately. NoisySigLIP's Table 1 results show that good image selection does not imply sensitivity to subtly incorrect descriptions; a single aggregate score can conceal this distinction.

Limitations & Future Work¶

The cost accounting is limited: aggregate denoising steps do not establish a halving of GPU wall-clock time. Cross-DiT experiments also require VAE decoding, whose overhead must be counted in deployment.
Early ranking creates an irreversible selection risk: a discarded trajectory might improve later. Retaining a small pool and pruning it progressively is a plausible extension, not a mechanism already validated in the body.
The evidence primarily uses synthetic trajectories and automatic evaluation. The body emphasizes basic prompts and points to the appendix for complete advanced-prompt analysis and failure cases; the current cache omits that appendix, so those results have not been verified here.
The fixed 20–29 interval depends on generation and scheduling settings. Cross-architecture transfer is demonstrated, but common calibration across samplers, step counts, and noise schedules requires further testing.
The paper attributes NoisySigLIP's R@1 difference to sigmoid versus softmax pretraining objectives. This is the authors' explanation; the available comparisons do not isolate that causal factor.

CLIP / SigLIP: The approach preserves dual-encoder similarity scoring while changing the visual branch's training distribution. The improvement primarily comes from noisy-state adaptation rather than introducing a larger language-based evaluator.
LatentCLIP: LatentCLIP brings CLIP capabilities into latent representations, whereas this paper emphasizes ranking under substantial mid-generation noise. The paper describes LatentCLIP as trained on clean latents, highlighting that latent compatibility and early semantic recognition are different problems.
VQAScore: VQAScore supplies external experimental evaluation. It is neither a required module for every NoisyCLIP selection nor the training supervision specified in the body.
Dynamic classifier-free guidance: Online feedback can also adjust guidance strength; this work primarily controls which candidate trajectories continue or terminate. Combining the two is a research direction, but the current results do not establish the benefits of such a combination.

Rating¶

Novelty: 4/5. Connecting a noise-adapted dual encoder to intermediate BoN pruning is well motivated, although its core components build on established contrastive learning.
Experimental Thoroughness: 4/5. Covers training ranges, two vision-language backbones, transfer across generator architectures, and compute trade-offs, but lacks complete latency accounting and a verifiable appendix in the available cache.
Writing Quality: 4/5. The path from method to system benefit is clear, though stage numbering and relative percentages versus absolute scores require careful interpretation.
Value: 4/5. Useful for diffusion inference budgeting and online alignment monitoring, with practical gains depending on scoring overhead and early-ranking reliability.