Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generation

Conference: ECCV 2026
arXiv: 2606.13971
Project Page: https://xiaomeng-yang.github.io/Prompt2Effect/
Code: Yes (provided on project page)
Area: Image Generation / Video Generation / Diffusion Models
Keywords: Image-to-Video Generation, Low-Rank Adaptation, Hypernetwork, Training-Free, SVD Normalization

TL;DR

Prompt2Effect proposes a weight-driven hypernetwork that explicitly utilizes frozen base model weights as conditioning and normalizes the LoRA parameter prediction space into an SVD energy-ordered format. It directly synthesizes effect-specific LoRA weights in a single forward pass, achieving zero-shot training-free effect control for I2V diffusion models. Inference requires only 3.3 seconds while achieving quality comparable to full LoRA fine-tuning.

Background & Motivation

Recently, Image-to-Video (I2V) diffusion models have achieved immense success in generating high-fidelity videos from a single image and text instructions. However, beyond conventional semantic control, there is a growing demand for "effect-level personalization"—rendering input images with specific visual styles or dynamic special effects (e.g., "polar bear falling from the sky," "skeleton revealing," "fisheye animation"). Such complex dynamic effects often need to be extracted from dozens of unstructured reference videos, making direct execution via inference-time conditioning computationally prohibitive. Consequently, the existing standard practice is to train an individual Low-Rank Adaptation (LoRA) module for each effect. This involves manually collecting and labeling around 50 effect videos, followed by thousands of gradient optimization steps. The cost of training a single effect is up to 56 GPU hours, making iteration and trial-and-error in creative workflows almost impractical.

To bypass the gradient overhead of per-effect training, hypernetworks have attracted attention as a solution to amortize adaptation parameters into a single forward pass. In the image generation domain, methods like HyperDreamBooth have demonstrated potential in identity injection. However, extending these techniques directly to I2V diffusion models faces two fundamental limitations. First, existing hypernetworks rely purely on semantic conditions (text or noise embeddings) to predict weights, remaining completely oblivious to the structural geometry of each layer's weights in the base model being adapted. When the model dimension scales up to the size of a Diffusion Transformer, this "blind prediction" becomes highly ill-posed. Second, the factorized form of the LoRA matrix \(\Delta W = BA\) is inherently unidentifiable: for any invertible matrix \(R\), \((BR, R^{-1}A)\) produces the exact same change \(\Delta W\). When the hypernetwork needs to simultaneously predict adapters for hundreds of layers, each layer has infinitely many equivalent solutions, causing optimization to wildly oscillate in a redundant space. Empirical results show that existing methods fail completely at this scale.

The core insight of this paper is: LoRA is not an absolute parameter but a structured adaptation delta defined on top of the base weight \(W_0\). Since the predictor needs to generate \(\Delta W\), it should first understand \(W_0\). Core Idea: Shift the hypernetwork from purely semantic-driven to weight-driven—utilizing the rows/columns of the frozen base weights as structural prior inputs, and predicting SVD-normalized LoRA factors \(US^{1/2}\) and \(S^{1/2}V^\top\) to eliminate factorization ambiguity. This stably synthesizes all LoRA updates of high-dimensional I2V models in a single forward pass, achieving training-free zero-shot effect control.

Method

Overall Architecture

The pipeline of Prompt2Effect consists of two stages: one-time hypernetwork pre-training and zero-shot inference. During pre-training, a 1.3B-parameter Transformer hypernetwork learns the mapping from textual effect descriptions to the entire set of LoRA weights. During inference, any new effect text requires only a single forward pass to obtain all LoRA updates across all layers, which are then applied directly to the frozen I2V backbone to generate videos with special effects. Internally, the hypernetwork comprises three core stages: weight-driven input construction, joint Transformer processing, and SVD-normalized weight prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Frozen Base Weights W₀<br/>+ Textual Effect Description"] --> B["Stage 1: Weight-Driven Input Construction<br/>W₀ Sliced into Row/Col Tokens<br/>→ Projection → Cross-Attention Compression"]
    B --> C["Stage 2: Transformer Backbone<br/>Compressed Weight Tokens + Layer/Module Embeddings<br/>+ Text Cross-Attention<br/>→ Effect-Injected Latent Presentation"]
    C --> D["Stage 3: SVD-Normalized Weight Prediction<br/>Cross-Attention Decompression<br/>→ Linear Heads → Â、B̂"]
    D --> E["Predicted LoRA Weights<br/>ΔW = B̂·Â"]
    E --> F["Zero-Shot Effect Generation<br/>or Adaptation Initialization<br/>→ 10× Accelerated Fine-Tuning"]

Key Designs

1. Weight-Driven Input Construction: Anchoring Adaptation Prediction via Row/Column Structure of Base Weights

Existing hypernetworks predict weights directly from semantic embeddings, which is equivalent to forcing the predictor to generate adaptations without understanding the "input-output coordinates" of each layer. However, since LoRA is inherently a delta relative to \(W_0\), a predictor blind to \(W_0\) must implicitly infer its structural geometry in a high-dimensional space. This work proposes explicitly feeding \(W_0\) itself as a conditioning input: for each layer \(l\), the rows of \(W_0^l \in \mathbb{R}^{d_{out}\times d_{in}}\) are sliced into row tokens (\(d_{in}\)-dimensional) and the columns into column tokens (\(d_{out}\)-dimensional). These are mapped to a shared \(d\)-dimensional space via two learnable projection layers and then concatenated to form the sequence \(X_W^l\). Inspired by classical CUR matrix decomposition, using the actual rows and columns of \(W_0\) as tokens allows the hypernetwork to directly access the intrinsic feature coordinates and cross-feature correlations of the layer.

Through SVD spectral analysis, the paper reveals a key phenomenon: the singular value energy of \(W_0\) is highly concentrated in the first few principal components, whereas the energy distribution of \(\Delta W\) is much wider, scattering across many low-energy singular directions of \(W_0\)—revealing a clear "compressibility gap." Consequently, full-rank row/column information of \(W_0\) must be utilized, rather than a truncated SVD approximation. The long sequence from the full-rank input is compressed into a fixed length \(n \ll d_{out}+d_{in}\) using cross-attention (with \(n=256\) learnable queries as \(Q\) and the original tokens as \(K, V\)), preserving structural details while managing the computational overhead of the subsequent Transformer.

2. SVD-Normalized LoRA Prediction: Eliminating Factorization Ambiguity to Stabilize Large-Scale Weight Synthesis

The factorized form of LoRA, \(\Delta W = BA\), poses an inherent unidentifiability: for any invertible matrix \(R\), \((BR, R^{-1}A)\) and \((B, A)\) generate the exact same \(\Delta W\). When the hypernetwork must simultaneously predict \(A^l\) and \(B^l\) for hundreds of layers, there are infinitely many equivalent solutions per layer, causing optimization to easily oscillate in the redundant space. To address this, the authors propose normalizing the prediction target of the LoRA updates into a canonical SVD form. Specifically, singular value decomposition is performed on the pre-extracted ground-truth LoRA matrix \(\Delta W^l\):

\[\Delta W^l = U^l S^l (V^l)^\top \quad \Rightarrow \quad B^{\star,l} = U^l (S^l)^{1/2},\quad A^{\star,l} = (S^l)^{1/2} (V^l)^\top\]

SVD enforces that \(U\) and \(V\) are orthogonal, and \(S\) is a diagonal matrix of singular values sorted in descending order, yielding a unique representation up to sign ambiguity. By splitting the energy equally between the two factors, the equivalence class of each \(\Delta W^l\) is collapsed into a unique \((U S^{1/2}, S^{1/2}V^\top)\), dramatically constraining the search space for the hypernetwork. Experiments demonstrate that SVD-normalized training curves converge smoothly, whereas the unnormalized version oscillates wildly and yields nearly double the final error. What distinguishes this approach is that while methods like PiSSA use SVD for LoRA initialization to improve gradient training, this work leverages SVD to normalize the regression target space for hypernetwork optimization—operating on a different level.

3. Joint Transformer Hypernetwork: Globally Coordinating Multi-Layer Adaptation

The hypernetwork is a Transformer featuring 14 layers, 32 attention heads, and a hidden dimension of 2048, which simultaneously processes information across all adapted layers. For each layer \(l\), the compressed weight tokens \(Q_W^l\) are summed with a learnable layer index embedding \(e_l\) and a module type embedding \(e_M^l\) to obtain the input \(Z^l\). The effect text is encoded as \(T_{eff}\) using a frozen text encoder. Instead of simple concatenation, text is injected via cross-attention with weight tokens serving as queries and text tokens as keys/values. This enables the Transformer to model cross-layer semantic consistency in self-attention layers (e.g., coordinating all layers to produce a "melting" effect pattern) while absorbing semantic guidance from textual instructions in cross-attention layers. After jointly processing all \(L\) layers, the weight prediction head decompresses the latent representation via cross-attention (with original \(W_0^l\) tokens as queries and the hidden representation \(X_\mathcal{E}^l\) as keys/values) to recover the original dimensions. Two lightweight linear heads then output \(\hat B^l \in \mathbb{R}^{d_{out}\times r}\) and \(\hat A^l \in \mathbb{R}^{r \times d_{in}}\), respectively. During inference, a single forward pass produces \(\{(\hat A^l, \hat B^l)\}_{l=1}^L\) for all layers.

Loss & Training

The training employs a Normalized Mean Squared Error (NMSE) loss, which computes the relative Frobenius error between the predicted factors and the SVD-normalized targets for each layer, averaged across all layers:

\[\mathcal{L}_{\text{LoRA}} = \frac{1}{2L} \sum_{l=1}^L \left( \frac{1}{N} \sum_{b=1}^N \frac{\|\hat A^l_b - A^{\star,l}_b\|_F^2}{\|A^{\star,l}_b\|_F^2 + \epsilon} + \frac{1}{N} \sum_{b=1}^N \frac{\|\hat B^l_b - B^{\star,l}_b\|_F^2}{\|B^{\star,l}_b\|_F^2 + \epsilon} \right)\]

This relative normalization makes the loss insensitive to scale variances across different layers and effects, preventing layers with larger magnitudes from dominating the training and enabling stable convergence of the hypernetwork.

Key Experimental Results

Main Results

Zero-shot tests on IID effects (within the 70 training effects) show that Prompt2Effect outperforms baseline models across all metrics and achieves quality comparable to full LoRA fine-tuning (1000 steps of independent optimization per effect). The HyperDreamBooth baseline fails completely at the scale of I2V (scoring zero on VLM metrics, outputting noise).

Method	CLIP Score	Aesthetic	VLM Effect	VLM Context	VLM Temporal	VLM Overall
Base I2V	24.23	53.36	15.99	14.21	12.37	42.57
HyperDreamBooth	22.34	35.30	0.00	0.00	0.00	0.00
Prompt2Effect	25.10	56.30	27.89	15.34	13.70	56.93
LoRA (Upper Bound)	25.06	57.60	27.68	15.82	14.07	57.57

On OOD effects (32 unseen effects), zero-shot prediction slightly underperforms full LoRA, but when leveraged as an initialization for just 100 steps of fine-tuning (Init-100), it matches the performance of 1000-step full LoRA, achieving a 10× acceleration.

Method	CLIP Score	VLM Overall
Base I2V	23.17	42.46
Prompt2Effect (Zero-Shot)	23.52	48.87
Init-100	25.82	56.33
LoRA-100 (100 steps from scratch)	25.11	52.28
LoRA (1000 steps, Upper Bound)	25.73	56.70

The paper also reproduces core results on the open-source Wan2.1 I2V-14B model, verifying that the method is model-agnostic: Prompt2Effect surpasses the LoRA upper bound on both CLIP Score and VLM Effect (CLIP 24.79 vs 24.68, VLM Effect 34.51 vs 34.16).

Ablation Study

Configuration	VLM Overall (IID/OOD)	Description
Full Model	56.93 / 48.87	Weight-driven + SVD normalization + Global Transformer
Ensemble (Blocked Hypernetwork)	51.29 / 45.11	Independent prediction per block, losing cross-layer coordination
Noise (Removing Weight Input)	46.14 / 43.58	Pure semantic embedding, zero structural prior
Compressed tokens n=128	49.75 / 45.01	Over-compression causes details loss
Compressed tokens n=256	56.93 / 48.87	Optimal trade-off
Compressed tokens n=512	56.22 / 48.03	Increased computation but saturated gains

Supplementary ablations further reveal that removing weight input causes the NMSE to converge to ~0.65; using a truncated SVD half-rank input improves it to ~0.35, while the full-rank \(W_0\) input yields the lowest error at ~0.30. This confirms that the predictive cues for \(\Delta W\) are indeed distributed across the broad spectrum of \(W_0\).

Key Findings

• Weight-driven design is a core contribution rather than a minor addition: Removing the \(W_0\) conditioning drops the VLM Overall score from 56.93 to 46.14, demonstrating that purely semantic conditions are wholly insufficient for guiding high-dimensional LoRA synthesis.
• SVD normalization resolves substantial optimization bottlenecks: Training without SVD normalization causes NMSE curves to oscillate and converge to higher errors, whereas the normalized variant converges smoothly. This validates that "unidentifiability" is a fundamental barrier to large-scale weight synthesis in hypernetworks.
• Global coordination outperforms block-wise predictions: A joint Transformer that processes all layers simultaneously significantly outperforms block-wise independent predicting ensembles, indicating that cross-layer semantic consistency is vital for synthesizing coherent effects.
• Zero-shot compositional control: The predicted LoRA weights can be directly linearly interpolated (\(0.5\Delta W_A + 0.5\Delta W_B\)), yielding naturally blended video effects. This suggests that the hypernetwork successfully learns a functional, structured LoRA space.

Highlights & Insights

• Weight-driven paradigm: Since LoRA is inherently an increment relative to \(W_0\), it is intuitive that the predictor should be aware of \(W_0\). Although straightforward, prior works on video models largely overlooked this. Explicitly encoding the "who adapts whom" structural relationship into the hypernetwork's inputs is key to resolving instabilities in large-scale weight synthesis.
• Novel utility of SVD: Tools like PiSSA / Make-a-LoRA employ SVD to improve LoRA training convergence, whereas this work uses SVD to normalize the regression target space—collapsing equivalent classes into a unique representation. This essentially strips equivariant symmetry out of the parameter search space.
• 60,000× Cost reduction: Reducing 56 GPU hours to 3.3 seconds with negligible quality degradation translates the LoRA creation process from "overnight waiting" into "real-time interaction," representing a substantial breakthrough for creative workflows.
• "Prediction as initialization" hybrid paradigm: When zero-shot results are not precise enough, the predicted weights can serve as a strong initialization that reaches full LoRA quality in only 100 fine-tuning steps (10× speedup). This hybrid "predictive-then-fine-tune" paradigm strikes an elegant balance between instant previews and high-fidelity final outputs.
• Cross-model generalization: Reproducing core findings on the open-source Wan 2.1 model confirms that this methodology is model-agnostic and holds robust transferability across different I2V backbones.

Limitations & Future Work

• Limited OOD generalization: Under extreme out-of-distribution effects, pure zero-shot predictions might lack precision, requiring Init-100 fine-tuning to recover details. The authors acknowledge that this limitation stems from the size and diversity of the hypernetwork's training dataset.
• Heavy front-loaded training cost: The hypernetwork itself requires pre-training for 4000 epochs on 32 A100 GPUs, alongside preparing ground-truth LoRAs for 70 effects (each compiled from roughly 50 videos through manual filtering). This significant upfront investment makes the approach most suitable for train-once, use-forever deployment scenarios.
• Global effect limitation: The model currently predicts layer-wide fully-connected LoRA updates, making localized spatial edits (e.g., "only melting the object on the left") unsupported. Incorporating mask conditioning is flagged as a future direction.
• Unexplored T2V architectures: The paper validates the approach solely within I2V contexts; scaling to Text-to-Video backbones remains unexplored.
• Hypernetwork-to-backbone size ratio: A 1.3B hypernetwork vs. an 11B backbone is around 12% overhead. Though acceptable, how the hypernetwork's parameter scale must grow when scaling to significantly larger backbones warrants future investigation.

Related Work & Insights

• vs HyperDreamBooth: Both utilize hypernetworks to predict adapter weights. However, HyperDreamBooth is purely semantic-driven, which suffers from unstable training and noisy outputs (0.00 VLM score) at the I2V scale. Prompt2Effect's weight-driven design and SVD normalization allow it to scale stably to high-dimensional video models.
• vs LoFA: LoFA is a recent pioneering work in weight-driven LoRA prediction, but it targets compact conditions (such as style/pose reference images) and directly regresses the raw, unnormalized matrix. This work provides deeper analysis and solutions for regularizing the hypernetwork's output space via SVD normalization.
• vs PiSSA / Make-a-LoRA: These methods use SVD to improve the training convergence of LoRA. In contrast, this paper uniquely applies SVD to normalize the regression target space to assist the hypernetwork. A critical difference is that PiSSA targets gradient optimization, whereas this work targets regression prediction.
• vs Inference-time Control Methods (Tune-A-Video / StyleMaster / ControlNet, etc.): These methods either require test-time optimization or rely on pixel-aligned conditioning, making it difficult to handle unstructured global dynamic effects (e.g., "skeleton revealing"). This work bypasses inference-time computation using a hypernetwork, achieving true zero-shot capability.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Simultaneously introduces weight-driven conditioning and SVD normalization to a LoRA-predicting hypernetwork, identifying and resolving 'unidentifiability' as the fundamental hurdle in large-scale hypernetwork weight synthesis.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across IID/OOD scenarios covering both zero-shot and fast fine-tuning; successfully reproduced on the open-source Wan model; rigorous ablation studies on input layouts, compression rates, and coordination strategies; highly convincing training curve analysis.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear problems and motivations, structured methodology explanation, well-substantiated rationale for SVD normalization, clear diagrams, and comprehensive presentation of experimental results.]
• Value: ⭐⭐⭐⭐⭐ [Slashes the adaptation overhead for effect-level I2V from 56 hours to 3.3 seconds, creating a monumental breakthrough for creative pipelines; the weight-driven + SVD-normalized paradigm offers valuable lessons for larger-scale architectures.]

Related Papers

• [CVPR 2026] Training-free Motion Factorization for Compositional Video Generation
• [ECCV 2026] ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
• [CVPR 2026] CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion
• [AAAI 2026] T-LoRA: Single Image Diffusion Model Customization Without Overfitting
• [CVPR 2025] K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs