LibraGen: Playing a Balance Game in Subject-Driven Video Generation

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Phantom-video/LibraGen
Area: Video Generation
Keywords: subject-to-video, subject consistency, LoRA merging, direct preference optimization, dynamic CFG

TL;DR

LibraGen adapts a video foundation model with 9,000 high-quality SFT samples, combining pairing-specific LoRA merging, complementary DPO, and dynamic CFG to preserve identity and generation capabilities, reaching a text alignment score of 3.594 in Table 1 on its in-house benchmark.

Background & Motivation

Subject-to-video (S2V) generates a new video from reference images and a text prompt without requiring a reference image to become the fixed first frame. This is more flexible than image-to-video, but requires the model to preserve the specified subjects while making them move naturally according to the prompt. A strong text-to-video model already knows motion, aesthetics, and language; adaptation should add subject conditioning without eroding these capabilities.

Existing training pairs have two main sources: in-pair references come from the target video itself, whereas cross-pair references retrieve corresponding subjects from other videos. In-pair data provide strong correspondence but encourage copying poses, backgrounds, and lighting, producing pasted-looking subjects, weak motion, or poor prompt adherence. Cross-pair data reduce this leakage but weaken reference-target correspondence and can lose identity details, motivating LibraGen's task-incremental learning perspective instead of optimizing reference similarity alone.

Core Idea: first improve the achievable quality through human-assisted data curation, then train adapters with complementary biases and combine their weights with time-dependent conditioning, so learning identity need not come at the expense of motion and prompt adherence.

Method

Overall Architecture

The inputs are a text instruction and one or more reference images; the output is a new subject-consistent video. The backbone contains low-resolution and super-resolution MM-DiT networks with interleaved spatial and temporal attention, producing 480P video that can be refined to 720P. Training proceeds through high-quality data curation, lightweight subject injection, and Tune-to-Balance post-training; inference conditioning rewrites the user's prompt into the training caption format and dynamically controls image and text guidance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Source videos and references"] --> Data["High-quality data curation"]
    Data --> Inject["Lightweight subject injection"]
    Inject --> Train["Tune-to-Balance<br/>post-training"]
    Train --> Infer["Inference conditioning"]
    Input["User prompt and references"] --> Infer
    Infer --> Output["480P video<br/>optional refinement to 720P"]

Key Designs

1. High-quality data curation: remove examples that would degrade the foundation model

Sources include Panda70M, an in-house collection of 3 million clips, online platforms, and synthetic content. AutoShot and PySceneDetect segment videos into 2–11-second clips; motion and aesthetics experts retain the top 5%, followed by manual ratings of motion coherence, amplitude, and aesthetics on a 1–5 scale, retaining an average score of at least 4. The resulting thousand-scale training set therefore comes from large-scale curation, not from merely collecting a few thousand arbitrary clips.

Subject detection obtains bounding boxes in keyframes, followed by cropping and matching against a retrieval bank. For cross-pair data, the pipeline retrieves the 15 most similar candidates per subject, excludes the source video, and randomly selects a reference; in-pair references come from the same video. A VLM and human reviewers reject ambiguous subjects, check cross-video identity consistency, and constrain in-pair subject scale to reduce background leakage. Thus, data preparation includes detection and cropping: the claim of not requiring subject segmentation applies to model conditioning and does not mean the entire workflow lacks preprocessing.

Captioning explicitly binds each subject to its reference rather than attaching an arbitrary sentence to a video. The pipeline describes each reference, uses subject-box-reference metadata to replace subject names in the video caption with refined reference descriptions, and checks the result manually. Hierarchical tags and shared-prefix encoding support category statistics and model-feedback-driven distribution updates; the paper does not provide a complete reusable tagging taxonomy.

2. Lightweight subject injection: reuse the VAE and DiT rather than adding a separate visual pathway

The same VAE encodes \(n\) reference images and \(f\) video latent frames. Reference latents are appended after video latents along the temporal dimension, distinguishing them from a fixed I2V first frame and avoiding temporal RoPE offsets for the original video tokens. The noisy input follows \(z_t=(1-t)z+t\epsilon\), where \(\epsilon\sim\mathcal{N}(0,I)\).

The model also receives zero-padded reference conditioning and binary reference flags, concatenated along the channel dimension to form \((f+n)\times(2c+1)\times h\times w\) inputs. Flags identify reference positions, while the conditioning tensor supplies clean reference information; the mechanism is therefore more than simply mixing noisy reference frames into the video. It requires neither an additional vision encoder nor a separate subject-segmentation conditioning branch.

Local temporal DiT windows may see only part of a reference subject, weakening identity information. The paper introduces all-gather during training and inference to expose complete reference-subject information to the windows; this is a conditioning-visibility implementation detail, not another identity encoder.

3. Tune-to-Balance post-training: two merging stages address different biases

Low-resolution SFT trains separate LoRA adapters on cross-pair and in-pair data, followed by linear interpolation of their weights. The cross-pair model supplies motion, visual quality, and prompt responsiveness, while the in-pair model strengthens identity; excessive in-pair weight still copies reference poses, so the chosen coefficient is \(l_{\mathrm{in\text{-}pair}}=0.15\). This preserves two adjustable parameter directions rather than simply mixing both data types in one training batch.

The merged SFT model still needs more targeted preferences. Consis-DPO generates a positive sample with the cross-pair SFT model, then retains the initial noise, sampling steps, prompt, and reference images while increasing reference-embedding RoPE offsets in temporal attention to produce an identity-degraded negative. Figure 6 compares offsets of 0, 1024, and 2048: identity details deteriorate while the broad structure and background remain relatively stable. Manual curation retains pairs with clearly better subject consistency in the positive, focusing optimization on identity rather than uncontrolled changes across several quality dimensions.

However, Consis-DPO positives remain model-generated videos, and their quality ceiling and preference hacking can harm motion. Real-Fake DPO instead takes high-quality target videos from the SFT data as positives and generates negatives with the in-pair SFT model under the same prompts and references. Reviewers retain negatives that preserve identity but have poor visual or motion quality, teaching the distinction between a recognizable subject and a stiff, pasted-looking rendition. Because the source collection also contains synthetic content, "Real" should not be interpreted as a guarantee that every positive is camera-recorded footage.

Both DPO branches start from the merged SFT model and train separate LoRA adapters before another merging stage; the reported Consis-DPO and Real-Fake coefficients are 0.5 and 0.1. The objective takes the difference between positive and negative log-probability ratios relative to a reference policy, applies a temperature-scaled sigmoid, and minimizes its negative logarithm. The reference policy is initialized from low-resolution SFT and updated after every iteration through an exponential moving average of the current policy, rather than remaining permanently frozen. Equation (4) is corrupted in the text extraction, so this note describes its semantics without inventing unverified diffusion likelihood-computation details.

4. Inference conditioning: establish motion and layout before refining reference details

Training captions explicitly bind reference subjects, whereas users often provide short action descriptions. The multi-stage prompt rephraser first describes each reference in detail, then extracts distinguishable coarse descriptions and integrates them with the user prompt, and finally adds relevant motion and scene details. The main text calls the first-stage model Qwen 3, but reference [2] is the Qwen2.5-VL technical report; this source inconsistency leaves the exact reproducible version unresolved.

Dynamic CFG addresses a separate temporal issue: early denoising mainly establishes text-guided layouts and motion trajectories, whereas later denoising emphasizes reference-driven textures, colors, and identity details. The paper assigns \(\omega_1\) to reference conditioning and \(\omega_2\) to text, combining unconditional, reference-only, and reference-plus-text predictions. Static guidance uses \(\omega_1=\omega_2=3.5\); along the denoising execution order, the dynamic schedule increases reference guidance from 1 to 4 and decreases text guidance from 5 to 1. Describing the schedule in execution order avoids incorrectly equating later denoising with increasing noise time \(t\) in the interpolation above.

Super-resolution also retains reference conditioning so that refinement from 480P to 720P does not erase identity. Training quadruples contain the high-resolution target, a downsampled and slightly noise-blurred low-resolution counterpart, the prompt, and references; the network is initialized with merged low-resolution SFT LoRA and fine-tuned with an exponential moving average. Dynamic CFG is not a cost-free parameter change: the authors report two additional velocity-function evaluations relative to the standard equal-modality-guidance case, increasing inference latency.

Loss & Training

SFT uses Rectified Flow and the noise sampling strategy in reference [13]. The 9,000 low-resolution samples comprise 5,500 cross-pair and 3,500 in-pair examples. In-pair training increases reference-image dropout to mitigate copying, while cross-pair training increases text dropout to encourage visual conditioning; these runs stop after 8,000 and 5,000 iterations, respectively, with exact dropout probabilities unspecified. Super-resolution constructs quadruples from 4,000 selected examples and trains for another 5,000 iterations. Consis-DPO and Real-Fake DPO use 2,000 and 1,600 preference pairs, respectively, with 4,000 iterations per branch. These counts exclude foundation-model pretraining; the available main text does not fully specify LoRA rank, learning rate, batch size, or hardware budget.

Key Experimental Results

Main Results

The in-house benchmark contains 200 cases, with 1, 2, 3, and 4 reference images accounting for 26%, 50%, 14%, and 10%, covering single and multiple subjects, animated IPs, people, and consumer products. Some reference images are uncropped and contain small subjects, also testing robustness to subject occupancy. All Table 1 metrics are higher-is-better, but their scales differ and should not be averaged directly.

Method (original Table 1)	MS	MQ	AES	IQA	VQ	TA
Vidu Q1	0.5373	0.9924	0.6491	71.82	2.795	3.315
Kling O1	0.4965	0.9865	0.6479	72.84	2.797	3.567
MAGREF	0.3830	0.9853	0.6356	70.36	2.354	1.784
Phantom	0.3844	0.9873	0.6410	69.35	2.373	1.998
LibraGen	0.5380	0.9930	0.6496	71.60	2.795	3.594

The text defines MS and MQ as Motion Smoothness and Motion Quality; AES and IQA assess aesthetics and image quality, while VQ and TA use visual-quality and text-alignment reward models. The first two numerical columns in cached Table 3 appear reversed relative to Table 1: its full configuration reads 0.9930 and 0.5380, whereas Table 1 reads 0.5380 and 0.9930. The comparison above preserves Table 1's order; the ablation below omits these two columns rather than silently repairing an uncertain mapping.

Ablation Study

The plus signs in Table 3 denote cumulative additions, not independent component removals; each GSB compares against the preceding configuration rather than the initial baseline.

Cumulative configuration (original Table 3)	AES	IQA	VQ	TA	GSB
Cross-pair SFT baseline	0.6420	71.87	2.794	3.504	N/A
+ in-pair LoRA, coefficient 0.15	0.6306	71.45	2.708	3.466	0.190
+ Consis-DPO, coefficient 0.5	0.6334	71.10	2.714	3.490	0.286
+ Real-Fake DPO, coefficient 0.1	0.6394	71.42	2.756	3.524	0.025
+ dynamic CFG	0.6496	71.60	2.795	3.594	0.020

Subject consistency is assessed through a net-win ratio, not an ordinary win rate:

\[ \mathrm{GSB}=\frac{G-B}{G+S+B}. \]

\(G\), \(S\), and \(B\) count cases where LibraGen is better than, similar to, or worse than the comparator. For example, Table 2 reports 0.700 against MAGREF with 4 reference images: this is a net-win ratio of 0.700, not necessarily an absolute win rate of 70%. Against Kling O1, the ratios for 1, 2, 3, and 4 references are 0.077, 0.080, 0.071, and 0.100; all are positive, but they cannot be added to the successive comparisons in Table 3.

Key Findings

• Table 1's TA of 3.594 exceeds Kling O1's 3.567 by 0.027, a reward-score difference rather than accuracy percentage points.
• IQA of 71.60 trails Kling O1's 72.84 and Vidu Q1's 71.82; VQ of 2.795 also does not exceed Kling O1's 2.797, so the method does not lead every metric.
• Consis-DPO yields a successive GSB of 0.286, but IQA decreases from 71.45 to 71.10; identity optimization is not harmless to every quality dimension.
• Dynamic CFG raises TA from 3.524 to 3.594 and AES from 0.6394 to 0.6496, supporting different roles for text and reference conditions across denoising stages.

Highlights & Insights

• Pair construction determines model bias. Separate in-pair and cross-pair training turns identity strength and generation freedom into adjustable parameter directions rather than relying only on one data-mixture ratio.
• RoPE offsets become a tool for constructing preference negatives, not merely an architectural adjustment. Holding other sampling conditions fixed helps isolate identity differences, though manual curation remains necessary.
• Real-Fake DPO does not reward reference similarity alone. Identity-preserving but low-quality negatives provide a separate signal for restoring motion and visual quality.

Limitations & Future Work

• The authors explicitly acknowledge increased latency from dynamic CFG, but the available text provides no timing or throughput figures; the gain should not be described as free.
• Thousand-scale curated data rely on million-scale sources, retrieval infrastructure, and human review; there is no quality-versus-quantity comparison controlling annotation budget.
• The in-house benchmark has only 200 cases, and the available text does not fully report evaluator counts, confidence intervals, or statistical significance, warranting caution for small differences.
• The paper refers to OpenS2V-Nexus results and further DPO/CFG analyses in Appendices A–C, but the local cache ends with references; this note does not invent those unavailable results.
• The motion-column and prompt-model inconsistencies require checking original typesetting, configurations, and training details for reproduction; transfer to audio-video models is a proposal, not an evaluated result.

Related Work & Insights

• vs Phantom: Both use cross-modal subject conditioning; LibraGen additionally manages identity-quality conflicts through pairing-specific LoRA merging and complementary DPO branches.
• vs MAGREF: The paper attributes MAGREF's sensitivity to small subjects to concatenating references into one frame; temporal reference injection is advantageous on this benchmark, but that explanation is not an independently controlled experiment.
• vs in-pair-only S2V adaptation: LibraGen retains strongly corresponding data at limited adapter weight and turns the resulting model's copying tendency into useful Real-Fake negatives.

Rating

• Novelty: 4/5. The main contribution combines complementary data, targeted preference construction, and adapter merging; RoPE-based negatives are particularly concrete.
• Experimental Thoroughness: 3/5. Commercial and open-source comparisons, reference-count breakdowns, and cumulative ablations are useful, but public-benchmark appendices are unavailable and statistical/cost reporting is limited.
• Writing Quality: 4/5. The methodological thread is clear, though column ordering, model-version naming, and corrupted equation extraction require additional verification.
• Value: 4/5. A practical low-data subject-adaptation recipe, conditional on a strong backbone and curated human-reviewed data; sample counts alone do not establish low total cost.