Skip to content

Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute

Conference: ECCV 2026
Paper: ECCV 2026
Area: Video Generation
Keywords: subject-driven video generation / zero-shot customization / dual-task learning / gradient orthogonality / video diffusion model

TL;DR

This paper decomposes subject-driven video generation (SDV-Gen) into two alternately optimized sub-tasks — identity injection learned from subject-image pairs and motion-awareness preservation maintained on 4,000 arbitrary videos — and fine-tunes CogVideoX-5B on 200K subject-image pairs plus 4,000 videos in 288 A100 hours (about 1% of the training compute of prior zero-shot baselines), yielding identity-faithful, motion-rich videos at inference with no per-subject tuning at all.

Background & Motivation

Subject-driven video generation aims to produce videos that preserve a target subject's identity while letting scene, motion, and context vary freely, which makes it a cornerstone of personalized content creation. Existing work splits into two camps. Per-subject tuning methods (DreamVideo, MotionBooth, Still-Moving, CustomCrafter) fine-tune a separate model or LoRA for every identity: faithful, but unscalable, since customizing a single new subject already costs roughly 200 A100 hours. Zero-shot methods (VideoBooth, Phantom, VACE, HunyuanCustom) instead train one model that generalizes to unseen subjects, which appears to remove per-subject adaptation — but only by moving the cost into training. They implicitly assume that both identity and motion must be learned from subject-video pairs, so they require millions of such pairs and spend 10K–210K A100 hours on fine-tuning (about 70K–210K for VACE, 10K–30K for Phantom). That barrier keeps most researchers out of the field.

A natural alternative is to learn identity from subject-driven image generation (SDI-Gen) data, i.e. subject-image pairs, and simply inherit motion priors from the pretrained T2V backbone: subject-image pairs carry strong identity cues, while the backbone already encodes temporal dynamics from large-scale pretraining. The authors show this shortcut does not work. Naive fine-tuning on subject-image pairs severely degrades inter-frame temporal coherence — motion collapses into inconsistent dynamics or into near-static, frozen clips — which indicates that identity and motion modeling interfere with each other, and that treating SDV-Gen as a single objective is fundamentally flawed. Cascading is no better: an SDI-Gen → I2V pipeline breaks down when the subject is small or occluded in the first frame, because the image-to-video model misreads the low-resolution subject and fabricates details, after which identity degrades progressively across frames (the "toy car approaching the camera" case in Figure 4).

The angle this paper takes is to decouple adaptation explicitly into two complementary sub-tasks — identity injection, which only consumes subject-image pairs, and motion-awareness preservation, which only consumes a small set of arbitrary videos — and to alternate between them with stochastic task switching, spending most steps on the image branch and a minority on the video branch. Core idea: adapting a T2V backbone into a zero-shot SDV-Gen model needs no subject-video pairs at all — split "identity" and "motion" into two objectives supervised by different data sources and switch between them stochastically, since their gradients rapidly evolve toward nearly orthogonal update subspaces, giving stable convergence without gradient surgery and without any per-subject tuning.

Method

Overall Architecture

The contribution is not a new network but an adaptation recipe for turning a pretrained T2V model into a zero-shot subject-driven video model using two small data sources. On the input side there are two datasets: subject-image pairs \((\mathbf{I}_{\text{ref}}, \mathbf{I}_{\text{out}}, P)\) from OminiControl's Subject-200K — two images of the same subject under different poses, viewpoints, or contexts, plus a text prompt — which teach identity; and about 4,000 arbitrary videos \((P_{\text{vid}}, V)\) (1% of Pexels 400K), which only serve to hold temporal dynamics in place. Each training step draws a uniform random number to pick a branch: roughly 80% of steps perform identity injection on subject-image pairs, 20% perform I2V reconstruction on arbitrary videos. Trainable parameters are limited to LoRA on a subset of attention and normalization layers that interact with reference-image tokens; the rest of the backbone stays frozen. At inference there is no per-subject adaptation whatsoever: given one reference image and one prompt, the model generates a video zero-shot.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference image + text prompt"] --> B["Stochastic task switching<br/>one branch per step"]
    B -->|"video branch (p = 0.2)"| C["Motion-awareness preservation<br/>I2V reconstruction of arbitrary videos"]
    B -->|"image branch, other 80%"| D["Identity injection<br/>subject-image pairs + CLS anchor"]
    C --> E["LoRA update<br/>frozen backbone"]
    D --> E
    E --> F["Gradient orthogonality<br/>why switching stays stable"]
    F --> G["Zero-shot inference<br/>reference image + prompt to video"]

Key Designs

1. Stochastic task switching: split identity and motion into two alternately optimized sub-tasks

This is the backbone of the paper. Since naive fine-tuning on subject-image pairs collapses motion, identity and motion clearly should not be optimized jointly in a single objective — but the fix here is neither a regularizer nor a reweighting of losses: each step simply optimizes one of them. Concretely, every step draws \(u \sim U(0,1)\) and picks a branch:

\[\mathcal{L}_{\text{total}}=\begin{cases}\mathcal{L}_{\text{vid}}, & u < p\\ \mathcal{L}_{\text{img}}, & \text{otherwise}\end{cases}\]

where \(\mathcal{L}_{\text{img}}\) is the identity-injection loss on subject-image pairs, \(\mathcal{L}_{\text{vid}}\) is the text-video / I2V loss on arbitrary videos, and \(p=0.2\) is chosen empirically (the supplement ablates it in more detail). Equivalently, the expected parameter update is \(\mathbb{E}[g]=(1-p)\,g_{\text{img}}+p\,g_{\text{vid}}\): 80% of updates come from the image branch and 20% from the video branch. The authors describe this as stochastic proxy replay of the video distribution — video samples appear rarely, but often enough to keep reminding the model that the temporal dimension still exists.

The design buys two things at once. First, stability: each step computes the gradient of a single task, so no two opposing objectives descend on the same parameters simultaneously. Second, compute: under MM-DiT attention the token count grows roughly linearly with the number of latent frames and the attention cost roughly quadratically, so at \(T=13\) a video step costs about 169× an image step; the expected per-step cost under switching is

\[\mathbb{E}[C]\approx(1-p)C_{\text{img}}+p\,C_{\text{vid}}\approx 0.8C_{\text{img}}+0.2\times169\,C_{\text{img}}\approx 34.6\,C_{\text{img}},\]

far below video-only fine-tuning. This is one of the direct reasons the "1% Compute" claim holds.

2. Motion-awareness preservation: an I2V objective on a few arbitrary videos to hold temporal dynamics

The video branch addresses a specific failure: after image-only fine-tuning the model barely moves at inference. Rather than a pure T2V objective, the authors deliberately adopt the I2V form — a reference frame index is sampled from the video and the model must reconstruct the whole clip conditioned on it. Two reasons justify this choice: I2V (reference image → video) matches the modality seen at inference, avoiding a train/test interface mismatch; and it isolates motion from identity supervision, so the video branch carries no subject identity information at all. Because the videos are arbitrary (no relation to the evaluation subjects is required), this branch is essentially a very light anti-forgetting mechanism: a handful of real dynamic samples re-anchor the motion priors the backbone already has.

Single-frame conditioning admits a degenerate solution — simply replicating the reference appearance across time, which is cheap and looks "faithful". Two nearly free regularizers close it off. First, random reference-frame sampling: the conditioning frame index \(i\) is drawn uniformly over the clip instead of being fixed to the first frame, so copying frame one is no longer a stable solution. Second, image-token dropout: with probability \(p_{\text{drop}}\) a subset of tokens in the reference-frame encoding is dropped, forcing the model to rely on its own temporal priors rather than on the conditioning input. Both changes are trivial, but they attack the cause of the degenerate solution directly, and the supplement ablates each of them.

3. Identity injection: anchoring subject appearance on the token and reference-image tokens

The image branch follows OminiControl: the reference image and the output image are VAE-encoded into \(X_{\text{ref}}\) and \(X_{\text{out}}\), the prompt is encoded by a frozen T5 into \(C_T\), and the MM-DiT is trained to reconstruct \(X_{\text{out}}\) conditioned on \((X_{\text{ref}}, C_T)\). Because the input is a single frame, the RoPE embeddings are truncated to match the token count of one frame, keeping positional semantics consistent with the video branch. As for adaptation scope, LoRA is applied only to the attention and normalization layers that interact with reference-image tokens, leaving the rest of the backbone frozen — this matters because the motion prior lives precisely in those untouched weights: the less they are modified, the less motion is lost.

One further small but effective design prepends a special <CLS> token to the prompt (e.g. "An <CLS> armchair in the living room"), explicitly anchoring identity in the conditioning stream instead of relying on the implicit interaction between reference tokens and text. Ablations show that <CLS> improves the identity metrics (CLIP-I, DINO-I) without degrading motion quality. It also pairs well with stochastic switching: identity travels along the <CLS>-plus-reference-token pathway, the motion branch never touches it, so the two responsibilities stay separated at the representation level as well.

4. Gradient orthogonality: explaining why switching is stable and gradient surgery unnecessary

Alternating two objectives invites the usual multi-task worry about gradient conflict, whose standard remedy is a Gradient Surgery-style projection that removes conflicting components to guarantee a non-negative inner product every step. This paper measures first, then decides. At each checkpoint the model is frozen and \(g_{\text{img}}(t)\) and \(g_{\text{vid}}(t)\) are computed on mini-batches from the two datasets with respect to the trainable parameters (LoRA plus task-specific normalization layers), and their cosine similarity \(\phi(t)\) together with the \(\ell_2\) norms is logged. On both CogVideoX-5B and Wan 2.2-5B, \(\phi(t)\) collapses into a narrow band around zero shortly after initialization and stays there, while both norms remain well above the numerical noise floor and of comparable magnitude after the initial drop. The two objectives therefore really do update nearly disjoint parameter subspaces — this is not the artifact of both gradients vanishing, which would also drive the cosine toward zero. Under a local second-order quadratic approximation the authors further give a proposition showing that the gradient inner product of the two objectives decays as \(C\rho^{t}\) (⚠️ this is an explanatory model; the paper explicitly states it is not a formal guarantee for non-convex DiT training — refer to the original paper).

Given that observation, Gradient Surgery becomes unnecessary overhead: it requires computing both gradients and performing an extra projection every iteration, roughly doubling gradient-computation cost, whereas stochastic switching computes one gradient per step and reaches the same end effect of suppressing conflict. The authors therefore keep the simple switching scheme as the default. Establishing that the conflict resolves itself before deleting the machinery that handles conflict is the most elegant piece of method-level argumentation in the paper.

Loss & Training

The two objectives are denoising reconstruction losses — \(\mathcal{L}_{\text{img}}\) on subject-image pairs and \(\mathcal{L}_{\text{vid}}\) on arbitrary videos — used alternately according to the switching rule with \(p=0.2\). Optimization uses AdamW with a learning rate of \(5\times10^{-5}\) and a cosine-with-restarts schedule in BF16 mixed precision. CogVideoX-5B is fine-tuned for 4,000 steps with batch sizes of 256 (image) and 32 (video) and LoRA rank 128; Wan 2.1-1.3B and Wan 2.2-5B are each fine-tuned for 5,000 steps with batch sizes of 192/16 and LoRA rank 32; LoRA dropout is 0.2 throughout. Measured training cost is about 288 A100 GPU-hours per backbone. On the data side, identity supervision uses the full Subject-200K, with 2,000- and 4,000-pair subsets for the data-efficiency study; motion supervision uses roughly 4,000 Pexels videos (1% of Pexels-400K). Two sampling-rate variants are trained: an 8 FPS model aligned with the original CogVideoX setting, and a 16 FPS model further trained on Pexels videos to improve motion smoothness at higher frame rates.

On the "1% Compute" claim: the figure refers to training (adaptation) compute — 288 A100 hours, measured against the training cost of prior zero-shot baselines — and is neither an inference cost nor a comparison against the per-subject cost of tuning-based methods. Concretely, it is about 2.8% of Phantom (≈10K–30K A100 hours) and 0.4% of VACE (≈70K–210K), which rounds to roughly 1%. At inference the method still runs the full iterative denoising sampling procedure; what it removes is the need to retrain for every new subject — per-subject tuning costs about 200 A100 hours per subject, whereas this model is trained once and incurs zero additional training cost for any unseen subject.

Key Experimental Results

Main Results

Evaluation protocol: 30 reference images are randomly selected from recent SDI-Gen benchmark sets and the standard DreamBooth dataset, 8 prompts are generated per image with GPT, giving 240 reference-image–prompt pairs; VBench metrics are reported — Motion Smoothness (temporal consistency), Dynamic Degree (amount of motion), CLIP-T (text–video alignment), CLIP-I, and DINO-I (subject fidelity). Ablations use a 120-video subset (4 prompts per image).

Table 1 is the main VBench comparison (Phantom and VACE use their publicly available Wan-2.1 1.3B versions):

Method Motion Smooth.↑ Dynamic Degree↑ CLIP-T↑ CLIP-I↑ DINO-I↑
OminiControl → I2V (SDI-Gen cascade) 98.40 47.92 33.06 72.09 52.61
BLIP-Diffusion → I2V 97.81 48.10 28.83 79.27 57.21
IP-Adapter → I2V 97.74 51.90 28.94 76.60 51.41
VideoBooth 96.89 53.33 29.59 65.65 34.17
Phantom-1.3B 98.70 67.08 33.50 73.00 53.55
VACE-1.3B 98.74 43.75 33.70 73.54 54.47
MAGREF-480P 98.86 65.09 33.00 71.15 48.50
Ours (CogVideoX-5B) 98.23 76.25 33.65 76.19 61.23
Ours (Wan 2.1-1.3B) 98.97 53.75 32.51 74.65 58.77
Ours (Wan 2.2-5B) 98.09 82.50 31.29 76.97 57.27

Table 2 reports OpenS2V-Eval v1.1 on the Single-Domain setting, where Total is the normalized weighted sum of the remaining scores:

Method Training Cost Total↑ Aes↑ Smooth.↑ Amp.↑ FaceSim↑ Gme↑ Nexus↑ Natural↑
Vidu 2.0 – 52.90 43.32 91.88 17.52 36.19 66.96 44.84 66.11
Pika 2.1 – 53.12 47.43 86.07 26.32 32.33 69.84 47.35 64.68
Kling 1.6 – 56.67 45.97 85.76 47.17 39.27 65.36 49.30 73.63
VACE-P1.3B ≈70K hrs 49.20 48.93 95.68 11.91 18.04 70.78 36.24 66.85
VACE-1.3B ≈70K hrs 51.13 49.41 95.42 22.51 22.37 70.87 38.34 68.33
VACE-14B ≈210K hrs 61.75 48.94 93.16 19.69 64.65 65.86 50.82 70.56
Phantom-1.3B ≈10K hrs 54.50 49.00 93.70 16.38 44.03 69.54 37.72 66.76
Phantom-14B ≈30K hrs 57.02 47.46 94.86 41.55 51.82 70.07 35.30 71.11
SkyReels-A2-P14B – 55.06 40.85 85.54 26.41 54.42 61.81 48.60 61.85
MAGREF-480P – 53.44 46.31 92.63 27.43 33.77 69.02 42.45 68.33
Ours (CogVideoX-5B) 288 hrs 50.05 45.40 93.90 19.38 18.05 70.53 41.23 68.52
Ours (Wan 2.1-1.3B) 288 hrs 51.45 43.80 93.68 17.55 13.90 71.12 45.21 75.65
Ours (Wan 2.2-5B) 288 hrs 56.84 44.70 84.97 14.38 48.78 61.98 49.97 71.30

Ablation Study

Table 3 compares three training strategies on the same 120-sample subset:

Training strategy Motion Smooth. Dynamic Degree CLIP-T CLIP-I DINO-I
Image-only (subject-image pairs only) 99.60 0.84 32.67 71.15 43.19
Two-stage (image-only then I2V) 96.04 81.51 28.96 84.73 76.13
Ours (dual-task stochastic switching) 98.45 69.64 32.69 77.14 62.88

Table 4 studies the scale of the subject-image data (same 120-sample subset; FE is flow entropy, measuring diversity of subject motion, and CS is copy score, measuring near-rigid repetition of the reference subject):

Config Subject-image pairs Motion Smooth. Dynamic Degree CLIP-T CLIP-I DINO-I FE↑ CS↓
Ours-tiny 2,000 98.72 60.71 33.38 74.72 57.03 54.73 45.27
Ours-mini 4,000 97.99 78.33 33.22 75.87 58.86 58.91 41.09
Ours 200K 98.45 69.64 32.69 77.14 62.88 63.77 36.32

The column alignment of the Ours-mini and Ours rows in Table 4 was reconstructed from the trends stated in the paper's prose (more pairs → higher CLIP-I/DINO-I/FE, lower CS) together with the Ours row in Table 3; ⚠️ refer to the original table for exact values.

Key Findings

  • "1% Compute" is about training, not inference: 288 A100 hours per backbone is fine-tuning cost, about 2.8% of Phantom and 0.4% of VACE; against per-subject tuning (≈200 A100 hours per subject) the additional training cost per unseen subject is zero. Inference still runs the full iterative denoising sampling process.
  • The image branch alone kills motion: image-only training reaches a Dynamic Degree of just 0.84 while its Motion Smoothness is the highest (99.60) — that smoothness is an illusion produced by near-static output, and it is exactly the failure mode this paper targets.
  • Sequential two-stage fine-tuning is the opposite extreme: Dynamic Degree jumps to 81.51 and CLIP-I is the highest (84.73), but Motion Smoothness falls to 96.04 with severe artifacts and identity forgetting; only dual-task switching holds both in a usable range (98.45 / 69.64).
  • Two directions of the data-scaling story: as few as 2,000 pairs already suffice to learn the basic identity mapping (DINO-I 57.03), but copy-paste artifacts appear frequently — identity is preserved yet the subject repeats a near-identical pose and layout across frames (in the pig-surfing example of Figure 7, the small-data model never turns its head or changes posture). Scaling to 200K raises CLIP-I/DINO-I and FE while lowering CS, indicating that broader subject-image coverage mainly buys high-frequency detail and motion diversity rather than "whether it works at all".
  • The gradient conflict really does resolve itself: on both CogVideoX-5B and Wan 2.2-5B the cosine similarity converges into a narrow band around zero shortly after the start, while both gradient norms stay within the same order of magnitude and never sink into numerical noise, ruling out the trivial "vanishing norms" explanation. Gradient Surgery — which needs both gradients plus a projection each step, roughly doubling cost — is therefore unnecessary.
  • The recipe transfers across backbones: CogVideoX-5B, Wan 2.1-1.3B, and Wan 2.2-5B all use the same setting, each at roughly 288 A100 hours. Wan 2.2-5B attains the best Dynamic Degree (82.50) and an OpenS2V Total of 56.84, close to Phantom-14B (57.02) despite two orders of magnitude less training compute, and above commercial models Kling 1.6, Pika 2.1, and Vidu 2.0; its relatively low Amp. (14.38) is attributed to the TI2V backbone of Wan 2.2.
  • Failure cases: the cascaded SDI-Gen → I2V pipeline breaks identity propagation when the subject is small or occluded in the first frame (Figure 4); this model still fails under strong stylization or extreme appearance changes, and faces rely on generalization since Subject-200K contains almost no human faces.

Highlights & Insights

  • Treating "where the training data comes from" as the core design variable: rather than inventing a module, the paper re-partitions the supervision signal — identity from existing large-scale subject-image pairs, motion from 4,000 task-agnostic ordinary videos. This removes the prerequisite of building a subject-video pair dataset for the whole field, which matters more than the method itself.
  • "Prove the conflict disappears, then delete the machinery that handles conflict": measuring cosine similarity jointly with gradient norms rules out the trivial vanishing-norm explanation, after which per-step single-objective switching replaces gradient surgery at about half the cost. This measure → explain → simplify pattern transfers directly to other multi-task adaptation settings.
  • Random reference-frame sampling plus image-token dropout: two near-zero-cost changes that precisely cut off the most common degenerate solution in I2V fine-tuning (copying the reference frame). Any model conditioned on a single frame while generating a whole clip can borrow them as-is.
  • The <CLS> token as an identity anchor: one extra token buys gains in CLIP-I/DINO-I with no motion penalty — a very favorable engineering trade.
  • Transferable idea: whenever paired data for capability A is scarce but A and B each have their own data source, try dual-task stochastic switching plus a gradient-orthogonality check (audio-driven generation, long-video consistency, multi-view consistency). Using arbitrary videos at 1% scale as anti-forgetting proxy data is essentially a minimal form of rehearsal, far lighter than memory buffers or continual-learning machinery.

Limitations & Future Work

  • The primary training dataset, Subject-200K, is largely object-centric with almost no human faces; face capability comes from generalization, and the model fails under strong stylization or extreme appearance change. A systematic evaluation on face subjects is missing.
  • Motion quality still lags: the Amp./Motion scores on OpenS2V are below leading SDV-Gen methods (the authors acknowledge room for improvement), and the Wan 2.2-5B variant reaches only 14.38 on Amp.
  • Evaluation is limited in scale: VBench uses only 240 videos, OpenS2V is reported for Single-Domain only with no Multi-Domain results, and all ablations are run on CogVideoX, so conclusions for the Wan family remain extrapolations.
  • The method depends on an already strong T2V backbone (motion priors come entirely from pretraining) and does not apply to weak backbones or from-scratch training.
  • Observations of my own: the video branch uses only about 4,000 Pexels videos, so motion-type coverage is bounded by their distribution, and the switching probability \(p=0.2\) is an empirical choice that may not be optimal at other data scales. Natural next steps include replacing the fixed \(p\) with a curriculum (more video early, more image later) or adapting the switching probability to the measured degree of gradient orthogonality.
  • The "1%" is measured against the training budgets of zero-shot baselines, yet the multi-backbone experiments imply a total of roughly 288×3 GPU-hours — this is not the same measure as the "200 hours per subject" of per-subject tuning, so cross-table comparisons deserve care.
  • vs VACE / Phantom: they also perform zero-shot SDV-Gen but rely on million-scale subject-video pairs (Phantom-Data, OpenS2V-Nexus) and spend 10K–210K A100 hours; this paper uses no subject-video pairs at all and 288 hours, surpassing them on VBench DINO-I and Dynamic Degree and nearly matching Phantom-14B on OpenS2V Total.
  • vs VideoBooth: also trained once and generalizing to unseen subjects, but still supervised by subject-video pairs and limited in quality (DINO-I 34.17, Motion Smoothness 96.89), whereas all three of this paper's backbones exceed 57 on DINO-I.
  • vs SDI-Gen → I2V cascades (OminiControl / BLIP-Diffusion / IP-Adapter): cascades often reach high CLIP-I (79.27 for BLIP) but weaker motion dynamics, and identity propagation breaks when the first-frame subject is small or occluded; this paper injects the subject condition directly into the video diffusion model for a better balance between fidelity and motion.
  • vs per-subject tuning (CustomCrafter / Still-Moving): those cost about 200 A100 hours per new subject, and CustomCrafter still shows noticeable shape/texture drift on the teddy-bear sequence while Still-Moving loses fine facial details such as whiskers; this paper is zero-shot at inference and more identity-faithful in the qualitative comparisons.
  • vs multi-task optimization (Gradient Surgery) and continual learning (replay / memory buffers): Gradient Surgery removes conflict by projection at the price of two gradient computations per step, and continual learning relies on replay buffers against forgetting. Per-step single-objective switching achieves a similar effect here, and the gradient measurements explain why that suffices — for adaptation problems with few tasks and naturally near-orthogonal subspaces, a simple schedule can be more economical than explicit conflict handling.

Rating

  • Novelty: ⭐⭐⭐⭐ No new module, but reformulating SDV-Gen as a dual-task problem of identity injection plus motion preservation and removing the subject-video pair dependency is a genuinely valuable reframing.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Three backbones, two benchmarks (VBench and OpenS2V), strategy and data-scale ablations, and gradient measurements; points off because ablations concentrate on CogVideoX, OpenS2V is Single-Domain only, and the evaluation set is just 240 videos.
  • Writing Quality: ⭐⭐⭐⭐ A clear argument chain (failure observation → decomposition → gradient evidence → removing redundant machinery) with restrained formulas; the main weakness is that the PDF's table column alignment is hard to read and requires checking against the prose.
  • Value: ⭐⭐⭐⭐⭐ It lowers the compute barrier of zero-shot SDV-Gen from 10K–210K A100 hours to 288 hours with a recipe that transfers across backbones, which is highly actionable for resource-constrained researchers.