Tuning-free Visual Effect Transfer across Videos

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: visual effect transfer, reference video conditioning, effect-aligned triplets, video editing, classifier-free guidance

TL;DR

RefVFX uses cross-content, shared-effect training triplets and joint video conditioning to transfer dynamic effects from a reference video onto another video or image without tuning for each new effect, achieving a 65.7% overall-match preference win rate against Lucy Edit on Neural V2V.

Background & Motivation

Adding a hat that remains present throughout a video and gradually turning a person into metal are different editing tasks. The former can largely be specified through the final appearance; the latter also requires specifying when the change starts, how it develops, and how the person continues moving. A text prompt can request a metallic transformation but struggles to convey the reference clip's precise pacing, lighting changes, and occlusions. First-and-last-frame constraints define endpoints without fully describing the intervening dynamics.

A reference video offers a more direct control signal, but also contains people, scenes, and original motion that should not necessarily transfer. The model must borrow the effect's temporal behavior without copying the reference person's clothing or body structure into the input subject. Existing static-reference methods primarily address identity or style, while per-effect LoRA approaches require renewed optimization for new effects. A more basic obstacle is supervision: natural footage rarely provides versions before and after an effect with the same motion, let alone another subject demonstrating that identical effect.

The paper therefore treats data construction and the conditioning interface as a joint problem, rather than merely attaching a reference encoder to an existing editor. Its data presents the same reusable effect across different content, while the model receives reference dynamics, input content, and textual semantics together. The title's tuning-free claim means that a trained model needs no parameter updates for a new reference, not that the system requires no training. Core Idea: teach the model to read transformation dynamics from reference videos using cross-content, effect-aligned triplets, then apply those dynamics to input content and motion through joint latent conditioning and multidirectional guidance.

Method

Overall Architecture

Inputs comprise a reference video demonstrating an effect, an image or video to edit, and a text prompt providing high-level semantics. The output is a new video whose scene and motion should come from the input and whose dynamic effect should come from the reference. Before training, effect-aligned data covers LoRA image-to-video effects, neurally generated video-to-video effects, and programmatic temporal effects. During training, joint latent conditioning exposes the diffusion transformer to both reference and input, supervised by a target video with the effect applied to the input. At inference time, the same trained model uses multidirectional guidance to adjust input preservation and reference adherence. The image editors, pose extractor, and video generators used to construct data belong to the offline supervision pipeline, not a mandatory sequence executed for every user edit.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Sources["LoRA / Neural Pairs / Code Effects"] --> Data["Effect-Aligned Data"]
    Data -.->|Training Supervision| Joint["Joint Latent Conditioning"]
    Inputs["Reference Video + Input Image or Video<br/>Text Prompt"] --> Joint
    Joint --> Guidance["Multidirectional Guidance"]
    Guidance --> Output["Effect-Transferred Video"]

Key Designs

1. Effect-Aligned Data: make shared effects, rather than shared subjects, the supervisory cue

The authors first define an effect category and collect paired results of applying it to several different inputs. For example, gradual metallic transformation can be applied to different people while the effect definition remains fixed. To construct a training example, they select two distinct input-output pairs under that effect, use the first pair's output as the reference, and use the second pair's input and output as the editing input and target. The first pair's original input is discarded, so the model cannot rely on an explicit before-and-after difference for the reference. This construction directly matches deployment: the user only needs to provide a reference video already containing the effect. The pairing relationship in Section 3.2 can be written as:

\[ \mathcal{T}_{i,j,\ell} =\bigl(\mathrm{output}_{ij},\mathrm{input}_{i\ell},\mathrm{output}_{i\ell}\bigr), \qquad j\ne\ell. \]

Here \(i\) identifies a shared effect, while \(j\) and \(\ell\) identify different content instances under that effect; the entries are reference, input, and target, respectively. This is a data-organization relationship, not a new loss, and does not imply that the method explicitly estimates a perfectly content-disentangled effect vector.

The first data source uses existing image-to-video LoRAs, treating each adapter as an effect. The same adapter is applied to multiple synthetic images to produce videos that can serve as references and targets for one another. The static model input is the target video's first frame, so this source primarily teaches generation of effect-bearing dynamics from static content. The main paper reports 43 LoRAs, approximately 300 videos per effect on average, and over 14K clips; these are approximate reported statistics, not exact counts recoverable through multiplication.

The second source addresses the harder problem of motion alignment: independently generating two similar videos does not ensure that they perform the same action. The pipeline first generates a subject image, then uses an image editor to change pose, viewpoint, and expression, producing an effect-free endpoint image. It edits that endpoint again to introduce the desired effect, such as turning the person into metal. The original first frame and effect-free endpoint are passed to a first-last-frame video model to generate a base video. The system extracts intermediate poses from that video, then generates an edited video conditioned on the original first frame, effect-bearing last frame, and intermediate poses. The two videos thus share motion constraints, while the edited version can transition over time from ordinary appearance to the desired effect. Pose constrains motion but does not guarantee identical texture pixels; interpreting these pairs as exactly pixel-registered would overstate supervision quality. GPT-4o expands the effect prompts, and Figure 4 illustrates this pairing pipeline.

The third source uses programmatic effects to supply reproducible, fine-grained spatiotemporal changes. Foreground-background masks from Senorita data allow the authors to select the full frame, foreground, or background as the affected region. They combine visual operators such as posterization, pixelation, and dithering with parameters such as color-bin count or pixel-block size. Temporal operators control wipe direction, onset, and duration, allowing the same appearance change to enter the frame gradually. Once the region, visual parameters, and temporal parameters are sampled, that identical configuration is applied to multiple videos to create genuinely shared-effect references and targets. This pipeline produces approximately 100K videos covering 1,500 synthetic effects; the complete dataset is reported to contain over 120K triplets and 1,700 effects. The sources respectively provide complex semantic transformations, motion-consistent video edits, and precisely controllable local dynamics, rather than repeatedly collecting the same supervision.

2. Joint Latent Conditioning: read input and reference in a shared attention space

The backbone is Wan2.1's first-last-frame-to-video model, which originally concatenates noisy latents, conditioning latents, and masks along channels. These tensors are patchified and embedded into tokens before the diffusion transformer predicts velocity. RefVFX places input-video latents in the conditioning branch, giving the model direct access to the scene, subject, and motion it should preserve. Encoded reference-video latents are concatenated along frame width beside both the noisy and conditioning branches, forming an adjacent reference region. This does not append the reference to the end of the temporal sequence; reference and output can exchange information through spatial self-attention at their temporal positions. The model can therefore use the reference's temporal changes while consulting the input to determine which content those changes should affect in the generated region.

The hybrid mask in Figure 5 further distinguishes conditioning frames that must be preserved from those that may be edited. Frames directly preserved in the output receive 1, editable frames receive 0.5, and the reference-region mask is entirely 1. These values specify conditioning roles and should not be interpreted as pixel opacity or proven fidelity probabilities. Reference latents appear in both branches, but the branches merge along channels before patch embedding, so they do not form two independently duplicated reference-token sequences. The authors report twice the token count of base generation, increasing accessible reference information at an additional computational cost. Text still supplies high-level semantics, while the videos convey details that are difficult to express through text alone. The main paper defers details such as cross-attention configurations to the supplement; the schematic alone does not specify a complete implementation.

3. Multidirectional Guidance: make input fidelity and effect strength adjustable

A single fixed conditioning strength cannot accommodate every real-video editing preference. A user may want a strong material transformation while preserving clothing and facial shape, or may prioritize an exaggerated deformation from the reference. During training, the authors independently drop reference, input-video, and text conditioning with low probability, allowing the model to predict under incomplete conditioning. At inference time, the system combines multiple classifier-free guidance directions for text, input video, and reference video, with latent normalization following prior work. Increasing input guidance generally preserves clothing and facial details, whereas increasing reference guidance strengthens both the effect and potentially entangled appearance cues. The muscle-shape and clothing changes in Figure 8 demonstrate an adjustable trade-off, not perfect separation of effect and identity. The supplied main paper does not contain the full guidance formula or conditioning-dropout probabilities, so this note explains the mechanism without inventing coefficients or equations. No inference-time optimization also does not mean one-step video generation: it eliminates per-example parameter updates while retaining diffusion generation and associated guidance computation.

A Worked Example

Consider Figure 4's metallic transformation: the offline pipeline first creates an ordinary subject's initial image and an ordinary endpoint with a changed pose. These ordinary endpoints generate a base motion video; the metallic endpoint and poses extracted from that video generate an approximately motion-aligned effect-bearing version. Repeating the effect for another subject and using the first subject's effect-bearing video as reference yields a cross-subject training triplet. At deployment, a user supplies a reference demonstrating metallic transformation and their own input video, without rerunning the image-editing and pose-pairing pipeline. Joint latent conditioning supplies both videos to the same generator: the reference describes the transformation, while the input identifies who moves in which scene. Multidirectional guidance then adjusts metallic-transformation strength and subject-detail preservation; this walkthrough illustrates the mechanism rather than adding a quantitative experiment.

Loss & Training

The authors LoRA-fine-tune Wan2.1's 14B first-last-frame-to-video model for 10K steps with a batch size of 8. Training uses a single node with 8 NVIDIA A100 GPUs and samples the three data sources equally, preventing the largest programmatic subset from dominating batches by size alone. The model predicts target-video diffusion velocity; the main paper specifies no additional standalone effect-disentanglement loss. Besides conditioning dropout, training occasionally supplies the true target last frame to retain the backbone's first-and-last-frame control capability. Offline data generation, unified model fine-tuning, and inference without per-effect tuning are therefore distinct stages; tuning-free does not imply zero training cost.

Key Experimental Results

Main Results

I2V testing includes 28 unseen LoRA effects and 56 reference videos. V2V validation contains over 100 instances drawn from unseen LoRAs, unseen neural-effect prompts, and unseen programmatic parameters, with Neural V2V and Code-Based V2V reported separately. Neurally paired validation examples are manually quality-filtered, so the results do not automatically represent unfiltered real-user videos. The user study uses Amazon Mechanical Turk two-alternative forced-choice comparisons, with over 19,000 annotations from over 500 unique annotators. RVA measures Reference Video Adherence, IVA measures Input Video Adherence, and OM measures Overall Match when applying the reference effect while preserving the input.

The following excerpt is from main-paper Table 1, page 9. Every entry is the full RefVFX model's win rate in percent against the row's opponent, with standard deviation, not that opponent's absolute score.

Opponent	Neural RVA	Neural IVA	Neural OM	Code RVA	Code IVA	Code OM
Lucy Edit	62.3 ± 2.9	60.7 ± 2.8	65.7 ± 3.1	64.7 ± 4.5	61.4 ± 5.0	65.3 ± 5.0
VACE (Depth)	58.3 ± 2.5	59.1 ± 2.6	66.8 ± 2.1	65.7 ± 5.4	56.2 ± 5.0	61.9 ± 5.0
VACE (Pose)	57.0 ± 2.1	53.7 ± 2.2	60.0 ± 2.2	62.9 ± 4.8	63.3 ± 4.9	64.3 ± 4.7

These opponents receive identical textual effect descriptions but cannot directly consume reference videos, so the comparison measures task-level systems rather than architectures given identical information. For I2V, Table 2 on page 10 reports RefVFX's RVA win rate of 57.1 ± 2.4% and OM win rate of 63.4 ± 2.4% against Wan2.1.

Ablation Study

The next excerpt also comes from main-paper Table 1, page 9, with the same direction: full-model win rates against ablated opponents. The two data ablations use the same number of training steps; V2V + I2V excludes programmatic data, while No Ref removes the reference video at inference from the fully trained model.

Ablated Opponent	Neural RVA	Neural IVA	Neural OM	Code RVA	Code IVA	Code OM
V2V Training	57.8 ± 2.4	53.9 ± 3.1	60.7 ± 3.2	66.2 ± 4.5	61.1 ± 5.0	66.2 ± 4.1
V2V + I2V Training	55.2 ± 3.2	52.6 ± 3.5	56.1 ± 2.8	55.5 ± 3.9	56.2 ± 4.0	57.5 ± 3.5
No Ref	57.5 ± 2.8	51.3 ± 3.1	67.2 ± 2.9	59.5 ± 3.5	54.3 ± 4.0	61.3 ± 3.9

With programmatic data included, the full model also achieves a 56.1% Neural V2V OM win rate against V2V + I2V, indicating benefits beyond programmatic-effect testing. Against No Ref, Neural IVA is nearly tied at 51.3%, while OM reaches 67.2%, supporting the reference's role in completing effect transfer rather than merely reproducing the input. These are preference comparisons, not equivalent percentage-point drops in absolute accuracy after removing a component.

Key Findings

Main-paper Table 4, page 15, uses VideoPrism video-embedding similarity to compare output with input and reference; higher values indicate closer embeddings. The validation set has no ground-truth target outputs, making these proxy metrics rather than direct measurements of effect isolation or motion fidelity.

Method	Neural Input Sim.	Neural Ref Sim.	Code Input Sim.	Code Ref Sim.
Wan VACE Pose	0.9068	0.6539	0.9225	0.6002
Wan VACE Depth	0.9460	0.6226	0.9394	0.5998
Lucy Edit	0.7544	0.6852	0.8882	0.6539
RefVFX	0.8568	0.7014	0.9479	0.7169

RefVFX has the highest reference similarity in both V2V settings, but lower Neural input similarity than both VACE configurations; it should not be described as winning every metric. Table 3 on page 15 likewise reports I2V reference similarity of 0.7378 for RefVFX versus 0.7230 for Wan2.1, but first-frame CLIP similarity of 0.7698 versus 0.7911. The authors interpret higher input similarity as possible under-editing, consistent with the human study and qualitative examples, but embedding scores alone cannot exclude alternatives such as identity drift. Some win-rate ranges summarized in the paper's prose do not cover every ablation row in Table 1; this note uses individual table entries rather than treating those summaries as exhaustive ranges.

Highlights & Insights

• The essential data unit is not an effect-bearing video, but a reusable pairing of the same effect across different content. Discarding the reference's original input aligns training with actual usage.
• The endpoint-plus-intermediate-pose pipeline assigns appearance change and motion constraints to different conditioning signals. It is better suited to editing supervision than independently generated pairs, while remaining dependent on generator quality.
• Programmatic effects add precisely specified temporal behavior, not just training volume. Their ablation benefit on neural effects suggests that simple, controlled supervision can also support complex editing.

Limitations & Future Work

• The authors acknowledge misuse risks in identity-related editing and plan model cards and acceptable-use terms; these governance measures do not technically eliminate inappropriate use.
• References may transfer subject appearance along with the desired effect, as Figure 8 shows clothing and muscle shape changing with reference guidance. Explicit identity-preservation evaluation and effect-region constraints warrant investigation.
• The main paper omits the full guidance formula, conditioning-dropout probabilities, latency, and peak memory, and the supplied text contains no supplement. Tuning-free does not establish real-time operation or justify unverified reproduction parameters.
• The human study provides means and standard deviations, but the main paper is insufficient to reconstruct grouped statistics and significance tests. IVA comparisons near 50% should not be declared significant from standard deviations alone.
• Dataset counts use different units and approximate reporting: 43 LoRAs, roughly 300 clips on average, and over 14K clips do not multiply to an exact match. Reproduction requires an actual release manifest rather than inferring missing subset sizes.

Related Work & Insights

• Compared with Dynamic Concepts: these approaches investigate dynamic-concept personalization, with some requiring per-instance LoRAs or limited training coverage; RefVFX reads new effects through reference conditioning, moving the cost to unified-model training and multi-source data construction.
• Compared with VACE: VACE provides pose, depth, and keyframe conditioning; RefVFX builds on conditional video modeling and adds a complete reference-effect video, making temporal transformation behavior a separate input.
• Compared with Lucy Edit: Lucy Edit edits videos directly through text; the paper observes static edits in its tested examples, whereas RefVFX supplies dynamic detail through references, without establishing that all text-based editing necessarily fails to express temporal changes.
• Research direction: controlled tests with identical effects across different subjects and motions could separately measure effect timing, identity leakage, and input-motion deviation. This is a reader-proposed evaluation direction, not an experiment already performed in the paper.

Rating

• Novelty: 4/5. Cross-video temporal-effect transfer is closely integrated with scalable effect-aligned supervision.
• Experimental Thoroughness: 4/5. Multi-task user studies, data ablations, and embedding metrics are included, but precise effect ground truth and complete statistical details are missing.
• Writing Quality: 4/5. The data pipeline and conditioning diagrams are clear, while some key parameters are deferred to the supplement and summary ranges require checking against tables.
• Value: 4/5. A useful direction for example-driven video editing, with training and inference cost and identity preservation still requiring further evaluation.