Skip to content

NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation

Conference: ECCV 2026
Paper: ECCV Full Text
Project Page: https://yujiangpu20.github.io/noiseasier/
Area: Video Generation
Keywords: Text-to-Video, Test-Time Scaling, Noise Optimization, Video Consistency Models, Multi-Objective Reward

TL;DR

NoisEasier introduces a test-time noise optimization framework based on Video Consistency Models (VCMs) that jointly refines initial latents and intermediate perturbations under multi-objective differentiable rewards and negative-aware calibration, substantially boosting fine-grained compositional video alignment without altering model parameters.

Background & Motivation

Diffusion models have achieved remarkable success in text-to-video (T2V) synthesis, generating videos with realistic visual appearances and coherent temporal motion. Nevertheless, current systems continue to struggle with fine-grained compositional alignment, frequently failing on multi-concept coordination tasks such as attribute binding, spatial relations, and complex object interactions. To align generative models with human preferences, existing works predominantly rely on reinforcement learning (RLHF / DPO) or offline reward-based fine-tuning on video generators.

However, offline reward fine-tuning presents two fundamental shortcomings. First, learned preferences are baked directly into the model parameters, making adaptation to unseen prompt distributions or updated reward objectives prohibitively expensive. Second, offline tuning remains prone to reward hacking, where optimizing imperfect reward functions often degrades visual fidelity or results in static, motion-deprived video outputs. While test-time noise optimization has emerged as a promising training-free paradigm in image generation, porting gradient-based latent refinement to video diffusion models is severely hindered by the excessive computational cost and memory footprint of backpropagating through long denoising trajectories, compounded by fragile spatiotemporal video rewards.

To resolve these computational and stability bottlenecks, this work leverages the rapid inference properties of Video Consistency Models (VCMs), which compress the long sampling process into just 4 steps, rendering backpropagation practical under realistic inference budgets. Core idea: expand test-time optimization from the initial latent alone to the entire stochastic trajectory (including all intermediate perturbations), supervised by a balanced multi-objective reward combining image-text, video-text, and camera-subtracted optical flow dynamics alongside LLM-generated hard negative prompt calibration.

Method

Overall Architecture

The inference pipeline of NoisEasier is built upon a pretrained Video Consistency Model (such as T2V-Turbo or AnimateLCM). Given a text prompt \(c\), an LLM is first utilized to construct a set of syntactically close yet semantically incorrect hard negative prompts. During the test-time refinement loop, the generator takes an initial Gaussian noise \(\mathbf{z}_T\) and intermediate perturbations \(\boldsymbol{\varepsilon}_t\) as inputs to produce a candidate video sequence across 4 consistency steps. The generated video is evaluated by a multi-objective reward suite comprising frame-level image-text alignment (HPSv2 / ImageReward), clip-level video-text alignment (ViCLIP), and a RAFT-based localized optical flow motion reward. The composite objective is contrastively calibrated against the negative prompts, and gradients are backpropagated to update all stochastic variables via AdamW.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Prompt c & LLM Hard Negatives"] --> B["Full Stochastic Noise Variables<br/>z_T and Intermediate Perturbations {ฮต_t}"]
    B --> C["4-Step Video Consistency Model<br/>Trajectory Integration & Video Synthesis"]
    C --> D["Multi-Objective Differentiable Rewards<br/>Image-Text + Video-Text + RAFT Local Motion"]
    D --> E["Negative-Aware Reward Calibration<br/>Contrastive Separation with Hard Distractors"]
    E -->|AdamW Gradient Ascent| B
    E -->|Reaches 25 Optimization Iterations| F["Final High-Fidelity Aligned Video"]

Key Designs

1. Full Stochastic Trajectory Optimization: Overcoming Initial-Noise Bottlenecks

Existing noise optimization techniques restrict the optimization parameter strictly to the initial latent \(\mathbf{z}_T\), limiting representation capacity and slowing gradient convergence. NoisEasier observes that Video Consistency Models inject Gaussian perturbations \(\boldsymbol{\varepsilon}_t \sim \mathcal{N}(0, \mathbf{I})\) at each step when transitioning between latents \(\mathbf{z}_{t-1} = \sqrt{\alpha_{t-1}}\,\hat{\mathbf{z}}_0 + \sqrt{1 - \alpha_{t-1}}\,\boldsymbol{\varepsilon}_t\). Rather than refining only \(\mathbf{z}_T\), NoisEasier jointly optimizes all stochastic variables \(\boldsymbol{\epsilon} = \{\mathbf{z}_T, \boldsymbol{\varepsilon}_1, \dots, \boldsymbol{\varepsilon}_T\}\). Because gradients for both initial noise and intermediate perturbations are computed in the exact same backward pass, this joint optimization introduces virtually zero additional computational or memory overhead. While initial noise governs global composition and macro layout, intermediate perturbations refine structural details and local styling, substantially accelerating convergence and extending the Pareto quality-cost frontier.

2. Complementary Multi-Objective Video Rewards: Counteracting Static Bias

Optimizing a single visual reward readily triggers reward hacking. Relying exclusively on image-text alignment models causes the generator to freeze motion in order to maximize frame-level alignment scores, while optimizing motion alone ruins semantic faithfulness. NoisEasier addresses this with a synergistic multi-objective formulation: - Frame-level image-text alignment: averaged ImageReward / HPSv2 scores across frames; - Clip-level video-text alignment: ViCLIP similarity averaged over two temporally offset 8-frame clips; - Localized optical flow motion reward: RAFT estimates adjacent frame optical flow \(\mathbf{F}_t \in \mathbb{R}^{2 \times H \times W}\). By subtracting the spatial mean flow \(\bar{\mathbf{f}}_t\), global camera translation is filtered out, leaving localized subject dynamics balanced against acceleration smoothness: $\(\mathcal{R}_{\text{motion}} = \tanh\left(\frac{1}{T-1}\sum_{t=1}^{T-1}\|\mathbf{F}_t - \bar{\mathbf{f}}_t\|\right) \cdot \left[1 - \tanh\left(\frac{1}{T-2}\sum_{t=1}^{T-2}\|\mathbf{F}_{t+1} - \mathbf{F}_t\|\right)\right]\)$ This term prevents the static video collapse induced by image rewards while enforcing coherent temporal continuity.

3. Negative-Aware Reward Calibration: Resolving VLM Bag-of-Words Degeneracy

Pretrained vision-language evaluators frequently act like bag-of-words models, insensitive to word order inversions, precise attribute bindings, and discrete counts. To introduce fine-grained discriminability, an LLM synthesizes \(N\) syntactically similar but semantically invalid hard negative prompts \(\{c^-_j\}_{j=1}^N\) for the target prompt \(c^+\). For a generated video \(x\) and base reward model \(\mathcal{R}\), the calibrated reward is defined as: $\(\tilde{\mathcal{R}}(x, c^+) = \mathcal{R}(x, c^+) - \tau \log \sum_{j=1}^N \exp\left(\frac{\mathcal{R}(x, c^-_j) - \mathcal{R}(x, c^+) + m}{\tau}\right)\)$ where \(m\) denotes a safety margin enforcing separation between positive and negative prompts, and \(\tau\) is the temperature scaling factor. This contrastive penalty suppresses superficial lexical activation and drives the optimization toward genuine compositionality in attribute bindings, spatial relations, and object counts.

Loss & Training

During the test-time phase, all underlying generator weights are kept frozen. The stochastic variables are updated via gradient ascent with AdamW using a learning rate of \(\eta = 0.01\), gradient clipping, and 25 optimization steps: $\(\boldsymbol{\epsilon}^{k+1} = \boldsymbol{\epsilon}^k + \eta \cdot \nabla_{\boldsymbol{\epsilon}} \left[ \sum_i \alpha_i \tilde{\mathcal{R}}_i(\mathcal{G}_\theta(\boldsymbol{\epsilon}^k, c^+), c^+) \right]\)$ To fit multi-step unrolled computational graphs within standard GPU VRAM budgets (e.g., RTX 6000 Ada), gradient checkpointing and mixed-precision inference are utilized throughout the pipeline.

Key Experimental Results

Main Results

NoisEasier is evaluated on VBench (measuring comprehensive semantic and dynamic quality) and T2V-CompBench (evaluating compositional binding capabilities). The main results across different short-step backbones are summarized below (corresponding to Table 1 and Table 2 in the paper).

Table 1: Quantitative Evaluation on VBench (%)

Backbone Model Overall Consist. Multiple Objects Color Spatial Relation. Motion Smooth. Dynamic Degree
ModelScope (Base) 25.67 38.98 81.72 33.68 95.79 66.39
VideoCrafter2 (Base) 28.23 40.66 92.92 35.86 97.73 42.50
AnimateLCM* 25.74 30.78 81.84 39.94 98.19 29.17
+ NoisEasier (Ours) 28.79 (โ†‘3.05) 44.04 (โ†‘13.26) 88.22 (โ†‘6.38) 46.70 (โ†‘6.76) 98.57 (โ†‘0.38) 30.56 (โ†‘1.39)
T2V-Turbo (MS)* 27.40 54.50 89.90 46.62 95.51 70.83
+ NoisEasier (Ours) 32.05 (โ†‘4.65) 76.62 (โ†‘22.12) 94.38 (โ†‘4.48) 48.93 (โ†‘2.31) 95.29 (โ†“0.22) 71.39 (โ†‘0.56)
T2V-Turbo (VC2)* 28.12 54.33 90.59 39.95 96.29 51.67
+ NoisEasier (Ours) 31.95 (โ†‘3.83) 79.18 (โ†‘24.85) 93.63 (โ†‘3.04) 52.61 (โ†‘12.66) 96.10 (โ†“0.19) 59.17 (โ†‘7.50)

Table 2: Quantitative Evaluation on T2V-CompBench

Model Configuration Consist. Attr. Spatial Action Interaction Numeracy
AnimateLCM* 0.6746 0.4651 0.3425 0.4504 0.2494
+ NoisEasier (Ours) 0.7807 (โ†‘0.1061) 0.5113 (โ†‘0.0462) 0.4362 (โ†‘0.0937) 0.5365 (โ†‘0.0861) 0.2567 (โ†‘0.0073)
T2V-Turbo (MS)* 0.7313 0.4620 0.4923 0.5822 0.3030
+ NoisEasier (Ours) 0.8375 (โ†‘0.1062) 0.5259 (โ†‘0.0639) 0.6477 (โ†‘0.1554) 0.6743 (โ†‘0.0921) 0.3983 (โ†‘0.0953)
T2V-Turbo (VC2)* 0.7852 0.5079 0.6241 0.7017 0.2900
+ NoisEasier (Ours) 0.8737 (โ†‘0.0885) 0.5660 (โ†‘0.0581) 0.7524 (โ†‘0.1283) 0.7915 (โ†‘0.0898) 0.3753 (โ†‘0.0853)

Ablation Study

Ablation studies confirm the necessity of each reward module and highlight the efficacy of full trajectory optimization (corresponding to Table 3 and Table 4 in the paper).

Table 3 & 4: Reward Components and Variable Ablation on T2V-Turbo (MS) / VBench

Experimental Variant Overall Consist. Multiple Objects Color Spatial Relation. Dynamic Degree Note
Baseline (T2V-Turbo MS) 27.40 54.50 89.90 46.62 70.83 Vanilla 4-step sampling
+ Single HPSv2 27.85 64.66 90.20 46.21 54.72 Dynamic degree drops sharply to 54.72
+ Single Motion Reward 26.05 47.68 87.32 42.07 85.56 Dynamics surge but semantic alignment degrades
+ Full Rewards w/o Negatives 31.74 75.82 93.29 47.95 73.33 High overall alignment but sub-optimal color/spatial
+ Initial Noise Only (+Init) 29.67 72.58 90.03 44.10 66.94 Decent gain, but inferior to full trajectory
+ Full Trajectory (NoisEasier) 32.05 76.62 94.38 48.93 71.39 Best overall performance across all dimensions

Key Findings

  • High Efficiency of Full Trajectory Optimization: As reported in Table 8, on an RTX 6000 Ada GPU, optimizing the initial noise of T2V-Turbo (MS) takes 44.63s (peak VRAM 22.93 GB), while optimizing all intermediate perturbations takes 45.28s (peak VRAM 22.95 GB) โ€” under 1.5% runtime increase and negligible memory overhead, yet yielding notable improvements (Multiple Objects jumps from 72.58% to 76.62%, Spatial Relation from 44.10% to 48.93%).
  • Complementarity with Offline Reward Fine-Tuning: Table 7 reveals that when NoisEasier is applied on top of an offline reward-finetuned (RFT) model, Multiple Objects improves from 73.49% to 84.56%, showing that test-time scaling unlocks additional alignment gains not fully amortized during offline training.
  • Superiority over Alternative Test-Time Strategies: Under the same compute budget, NoisEasier achieves higher Overall Consistency and Color binding compared to Best-of-n sampling (BoS) and LLM-based Prompt Enhancement (PE).

Highlights & Insights

  • Stochastic Perturbations as Free Optimization Variables: Rather than treating intermediate noise additions as passive sampling artifacts, recognizing them as active optimization variables expands trajectory control without additional forward/backward passes.
  • Negative Calibration Counteracts VLM Blind Spots: Formulating contrastive margins against LLM-crafted hard negative prompts provides an effective, model-agnostic recipe to cure the bag-of-words tendencies of vision-language evaluators.
  • Validating Test-Time Scaling for Video: Demonstrates that inference-time compute allocation via gradient-based search is an effective orthogonal dimension to scaling parameters or dataset sizes for video synthesis.

Limitations & Future Work

  • Bottlenecks in Temporal State Transitions: As shown in failure cases (e.g., ice melting into water), the method fails on long-term physical phase transitions because current lightweight reward models evaluate static frame semantics rather than causal temporal evolution.
  • Upper Bound on Complex Numeracy: While numeracy scores improve, scenarios with multiple distinct groups (e.g., "three apples and three birds") still exhibit count errors due to coarse VLM attention maps.
  • Inference Latency in Interactive Applications: Requiring 45โ€“177 seconds per clip prevents real-time deployment, suggesting future explorations into surrogate low-rank gradient estimation or accelerated backward passes.
  • vs DOODL / ReNO: DOODL optimizes across full 50-step diffusion graphs, resulting in tens of minutes of latency; ReNO is restricted to one-step image models. NoisEasier successfully scales test-time noise optimization to video models by leveraging short-step VCMs.
  • vs Video-RLHF / Direct Preference Optimization: Offline preference tuning locks alignment into weights and risks motion degradation via reward hacking; NoisEasier operates at test time, requires no retraining, and complements offline aligned backbones.

Rating

  • Novelty: โญโญโญโญ [Pioneering full-trajectory test-time noise optimization on short-step video consistency models]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across VBench, T2V-CompBench, ablations, efficiency profiling, and MTurk user studies]
  • Writing Quality: โญโญโญโญโญ [Clear motivation, structured methodology, clean mathematical formulation, and high-quality figures]
  • Value: โญโญโญโญ [Provides a practical, training-free test-time enhancement framework for fine-grained video generation]