ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA

Conference: ECCV2026
Paper: ECCV Paper
Project: https://id-lora.github.io
Area: Video Generation
Keywords: joint audio-video generation, identity preservation, In-Context LoRA, negative temporal positions, identity guidance

TL;DR

ID-LoRA uses a shared audio-video backbone conditioned on a first frame and reference speech, separates reference from target through negative temporal positions, and strengthens vocal identity through identity guidance, reaching 0.477 speaker similarity on cross-video evaluation while allowing text to change the scene and speaking style.

Background & Motivation

Personalized human video requires not only a recognizable face but also a recognizable voice, with both situated in a credible shared scene. Conventional cascaded systems clone speech first and then generate lip motion conditioned on that speech. This order suits animation driven by an existing soundtrack, but can carry a reference recording's quiet background and neutral delivery into the new video. Even when the prompt places the person on a noisy street, the already synthesized speech cannot respond to the depicted actions and acoustics. Audio-only models can add text control, but still cannot see when actions such as knocking or clapping occur in the video.

Joint audio-video models offer a different starting point: the modalities exchange information during generation and share scene-text conditioning. However, synchronized sound generation does not automatically preserve a specified speaker, and the foundation model lacks an adaptation mechanism for using a short recording as identity context. Same-video dubbing and editing methods can place reference and target at corresponding temporal coordinates because they modify the same underlying event. Here, the reference may come from an entirely different video, so shared positions risk confusing what was said in the reference with what should be generated now.

The paper therefore learns to use reference identity on top of LTX-2 rather than training a large audio-video model from scratch. The first frame anchors facial appearance, reference audio supplies vocal identity, and text specifies target content, environment sounds, and delivery. Zero-shot means that a trained model needs no per-person optimization for a new subject, not that the method requires no training. Core Idea: make reference audio readable context without aligning it to the target timeline, then amplify its identity contribution during sampling so that vocal identity and a new acoustic scene can coexist.

Method

Overall Architecture

Inputs are a first-frame image of the target person, reference audio, and text describing speech, actions, scene, and speaking style. The output is jointly generated video and audio, rather than completed speech followed by silent-video generation. The LTX-2 backbone is a 48-layer asymmetric dual-stream DiT with a 14B video stream and a 5B audio stream linked by bidirectional cross-modal attention. Its video VAE applies 32-fold spatial and 8-fold temporal compression; its audio VAE encodes mel spectrograms into latents. The video stream retains first-frame conditioning, while the audio stream prepends reference latents to the target audio latents being generated. Importantly, the method does not train an additional face-identity encoder or treat a complete reference video as a motion trajectory to reproduce. Audio in-context adaptation, negative temporal positions, and identity guidance respectively address reading identity, distinguishing the two audio sequences, and preventing identity dilution.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Reference audio + first frame<br/>Target text"] --> CONTEXT["Audio In-Context Adaptation"]
    CONTEXT --> POSITION["Negative Temporal Positions"]
    POSITION --> DENOISE["LTX-2 joint denoising<br/>Bidirectional audio-video attention"]
    DENOISE --> GUIDANCE["Identity Guidance"]
    GUIDANCE --> OUTPUT["Repeat sampling and decode<br/>Target video + audio"]
    TRAIN["Training: reference and target<br/>from the same speaker"] -.->|"Target denoising loss trains LoRA"| CONTEXT

Solid arrows describe condition assembly and sampling operations for an inference iteration; the dashed arrow indicates how training pairs teach the adaptation parameters. Identity guidance operates during repeated denoising, not as a voice-conversion step after audio decoding. Unified generation means avoiding an audio-to-video cascade; it does not mean a single network forward pass. The implementation retains LTX-2's two-stage rendering pipeline and compares predictions with and without the reference for guidance.

Key Designs

1. Audio In-Context Adaptation: let the generator read the reference voice directly

An audio VAE first encodes the reference recording into a temporally structured latent sequence. This sequence is concatenated with target audio latents along the sequence dimension for attention to access, rather than reduced beforehand to a fixed speaker label. The video stream continues generating from the first frame and text, so the supplied image remains the principal visual identity anchor. Existing cross-modal attention links the streams, allowing the evolving picture and sound to constrain one another. In-Context LoRA learns how to use identity supplied in the input rather than storing a separate set of weights for each test subject. Changing subjects therefore primarily involves changing the reference image and recording, without fine-tuning for that person.

This organization assigns distinct roles to the conditions: reference audio specifies who speaks, text specifies what and how, and the first frame specifies the person and initial visual scene. For an excited outdoor delivery, the model should extract the reference timbre without copying the recording studio's background as part of the identity. Training preprocessing therefore includes source separation of reference audio to reduce environment leakage. This does not add an environment classifier; it reduces identity-background confusion in the conditioning data itself. Adaptation with approximately 3K training pairs depends on the backbone's existing audio-video generation capabilities and should not be interpreted as learning the entire task from scratch with that amount of data.

2. Negative Temporal Positions: keep the reference accessible without occupying the target timeline

Concatenation alone does not specify the relationship between the audio sequences, and positional encoding further influences which segments attention associates. Corresponding reference-target positions help local alignment in same-video editing, but cross-video references lack a moment-by-moment correspondence. Reusing that coordinate relationship can encourage the model to treat reference content as audio that belongs at the current target moment. The paper assigns reference audio to negative temporal positions while retaining the target's pretrained nonnegative positions. Video uses 3D RoPE and audio uses 1D RoPE; the change concerns temporal positions of reference audio. Equation (3) and its accompanying description on page 7 specify the intervals:

\[ t^{\mathrm{ref}}\in[-T_{\mathrm{ref}},0),\qquad t^{\mathrm{target}}\in[0,T_{\mathrm{target}}]. \]

Here, \(T_{\mathrm{ref}}\) and \(T_{\mathrm{target}}\) denote the respective audio sequence lengths. The reference retains its internal ordering and relative temporal structure, preserving cues such as speaking rhythm. Target coordinates are not shifted to make room for the reference, avoiding a change to the positional distribution familiar to the backbone. Separation means distinct positional coordinates, not masking the reference out of attention or guaranteeing orthogonal representations. The model must still read the reference without interpreting that access as an instruction to align and replay the original utterance.

3. Identity Guidance: amplify reference-specific speaker features during sampling

Even after the model learns to read a reference, its strong generic speech prior may steer denoising toward a common voice rather than the requested one. Inspired by classifier-free guidance, the method compares predictions with and without reference conditioning and extrapolates along the reference-induced difference. Ordinary text CFG emphasizes text conditioning; identity guidance emphasizes timbre, rhythm, and pronunciation associated with reference audio. Text continues to govern scene content and environment sounds, so the goal is not to pull every acoustic property back toward the recording. The operation applies to the audio stream as an inference-time enhancement and does not require gradients from a separate speaker classifier.

Negative temporal positions and identity guidance should be understood together: the former clarifies reference conditioning, giving the latter a more reliable conditional difference to amplify. If reference and target remain entangled, amplifying reference influence can also amplify erroneous content dependencies. This helps explain why identity guidance alone slightly improves speaker similarity yet raises ablation WER to 0.252. The default identity guidance scale is 4.0, but the supplied main text does not contain the full numerical scale sweep. Equation (5) is corrupted in the extracted text, so this note describes the explicitly stated two-prediction extrapolation mechanism without reconstructing its exact algebra.

A Worked Example

Following Figure 1, a reference recording provides a woman's voice and a first frame supplies her appearance in the target scene. The prompt requests excited speech with a jackhammer in the background; this is an illustration of the mechanism, not an additional experiment. The recording is encoded and placed at negative temporal positions, while target audio and video start from their respective latents to be denoised. Audio attention reads vocal identity from the reference, text controls delivery and background sound, and the first frame constrains appearance. Joint attention couples lip motion with sound, while identity guidance strengthens reference-speaker differences throughout sampling. Decoding produces video and audio of the same scene without first fixing a scene-independent voice track. The intended distinction is to preserve who she is while allowing where she is and how she speaks to change.

Loss & Training

Training pairs reference audio with target audio-video from the same speaker and teaches target-latent denoising conditioned on the reference and text. The main text describes a standard diffusion denoising objective, without introducing an additional face-recognition or speaker-contrastive loss. Equation (4) is also corrupted in the extracted mathematical layout, so its damaged expression is not reconstructed as the authors' exact formula. Separate checkpoints are trained on CelebV-HQ and TalkVid, with segmentation, captioning, speaker clustering, and reference source separation during preprocessing. LoRA rank is 128; AdamW training runs for 6,000 steps on one NVIDIA H100, with learning rate \(2\times10^{-4}\) and batch size 4. First-frame conditioning is used with probability 0.9; the main text reports approximately 3K training pairs without itemizing final pair counts for each checkpoint here. Inference uses 30 denoising steps with video CFG, audio CFG, identity guidance, and AV-bimodal CFG scales of 3.0, 7.0, 4.0, and 3.0, respectively. STG uses scale 1.0 at block 29; outputs have 1024×1024 resolution, 121 frames, and 25 fps. These sampling settings condition the reported results, and single-GPU training should not be equated with low-latency deployment.

Key Experimental Results

Main Results

Automatic evaluation contains 120 video pairs from 63 held-out speakers: 36 pairs and 14 speakers in CelebV-HQ easy, 35 pairs and 8 speakers in hard, and 49 pairs and 41 speakers in TalkVid. Easy uses same-video pairs, whereas hard uses cross-video pairs to better test identity preservation when reference and target acoustic conditions differ. Spk Sim is cosine similarity from WavLM+ECAPA-TDNN features; Face Sim is a 5-frame average of ArcFace similarities, with higher values preferred for both. Lower LSE-D and higher LSE-C indicate better SyncNet synchronization metrics; higher CLAP indicates better audio alignment with environment and speaking-style prompts. WER uses Whisper-large-v3 transcription, with lower values preferred; the table retains the paper's decimal representation. The following excerpt comes from Table 1 on page 10; the two TalkVid rows use the in-domain checkpoint and the CelebV-HQ checkpoint without TalkVid fine-tuning, respectively.

Test Split	Method	Spk Sim ↑	Face Sim ↑	LSE-D ↓	LSE-C ↑	CLAP ↑	WER ↓
C-Hard	CosyVoice 3.0 + WAN2.2	0.391	0.890	11.40	1.50	0.249	0.362
C-Hard	VoiceCraft + WAN2.2	0.344	0.892	10.60	1.33	0.258	0.427
C-Hard	ElevenLabs + WAN2.2	0.357	0.894	11.86	1.72	0.238	0.154
C-Hard	Kling 2.6 Pro	0.385	0.854	9.49	3.47	0.316	0.121
C-Hard	ID-LoRA	0.477	0.874	8.49	3.90	0.363	0.113
TalkVid	ID-LoRA	0.599	0.772	10.62	3.09	0.385	0.054
TalkVid	ID-LoRA, Celeb → TalkVid	0.595	0.767	10.32	3.12	0.412	0.065

On hard, the speaker-similarity advantage over Kling is 0.092, approximately 24% relative; the advantage over the strongest cascaded speaker-similarity baseline is 0.086. The cross-dataset checkpoint loses only 0.004 similarity against the in-domain model and improves CLAP, but WER rises from 0.054 to 0.065, so not every metric improves. Table 2 on page 11 controls the video backbone: CosyVoice 3.0 + LTX reaches hard Spk Sim of 0.409, still below ID-LoRA's 0.477. In the same table, LTX-Zeroshot obtains hard LSE-D of 7.89 but WER of 2.393, showing why synchronization distance alone should not select the model.

Ablation Study

Table 3 on page 12 reports a full \(2\times2\) ablation on CelebV-HQ hard, toggling negative temporal positions and identity guidance. The table below preserves its LSE-D of 8.50; Table 1 reports 8.49 for the full model, and the paper does not explain the small discrepancy.

Config	Spk Sim ↑	Face Sim ↑	LSE-D ↓	CLAP ↑	WER ↓
Neither component	0.430	0.873	8.84	0.373	0.188
Negative temporal positions only	0.438	0.875	8.71	0.364	0.113
Identity guidance only	0.441	0.872	9.02	0.359	0.252
Negative temporal positions + identity guidance	0.477	0.874	8.50	0.363	0.113

Key Findings

• Individually, the components improve similarity by 0.008 and 0.011; together they improve it by 0.047, supporting a reinforcing rather than simply additive interaction.
• Negative temporal positions dominate intelligibility gains: they alone reduce WER from 0.188 to 0.113; adding identity guidance retains 0.113 while further improving identity similarity.
• CLAP does not increase with added components: it is 0.373 without them and 0.363 for the full model, so the main ablation benefits concern identity and speech content.
• Figure 3 and its discussion on pages 13–14 report voice-similarity preference against Kling of 73.1% versus 20.0%; the win rate after excluding ties is 78.5%, a different denominator.
• The physical-interaction MOS study uses 10 scenarios, 5 speakers, 50 paired samples, and 9 annotators per item; Figure 5 and the text on pages 14–15 report overall scores of 3.05 versus 2.90 and wins in 8/10 scenarios, still preliminary evidence.

Highlights & Insights

• Separating identity references from the target timeline is more fundamental than simply increasing conditioning strength. Inference-time amplification becomes more reliable once the condition has an unambiguous role.
• Existing cross-modal attention supports acoustic scene control without requiring a dedicated new architecture. Approximately 3K pairs demonstrate inexpensive adaptation of a strong backbone, not that its pretraining data can be dispensed with.
• Jointly considering identity, intelligibility, synchronization, and scene alignment prevents single-metric misjudgments. The LTX-Zeroshot analysis particularly warns that nearly static lips can also produce a low synchronization distance.

Limitations & Future Work

• The authors focus on single-speaker personalization and explicitly leave multi-speaker, cross-lingual, and general audio-visual reference-to-video generation to future work.
• This note's assessment: hard contains only 8 speakers, insufficient to infer robustness across all accents, recording devices, and complex acoustic conditions.
• Face Sim is not uniformly superior; the authors partly attribute this to realistic articulation affecting ArcFace, but that does not establish every lower score as merely metric bias.
• Physical-interaction MOS of 3.05 versus 2.90 supports improvement within limited scenarios, not a general understanding of physical acoustics.
• The main text lacks a complete comparison of inference time and memory, and identity guidance adds prediction branches; deployment cost requires separate measurement.
• The supplied text ends with the main paper and references, without supplementary long-clip tests, identity-guidance sweeps, or detailed facial-motion analysis; no missing results are invented here.

Related Work & Insights

• vs In-Context LoRA [17]: inherits reference-target latent concatenation but extends adaptation to vocal identity in joint audio-video generation and resolves positional ambiguity in cross-video references.
• vs Just-Dub-It [1] / EditYourself [11]: those edit existing video, whereas this method generates a new scene from reference audio, a first frame, and text; whether temporal correspondence exists determines the appropriate positional design.
• vs cascades using CosyVoice 3.0 [8] / VoiceCraft [26]: cascades fix speech first, while this method generates picture and sound together; the same-backbone comparison in Table 2 supports that the difference is not solely WAN versus LTX model family.
• vs DreamID-Omni [14] / MM-Sonate [29]: the main text cites approximately 1M and 100M training samples, respectively; different task scopes and backbones prevent converting these counts directly into an equal-quality efficiency advantage.
• Future experiments could test identity preservation across recording devices for the same speaker while separately measuring timbre transfer and reference-environment leakage; this is a suggestion from this note, not a reported experiment.

Rating

• Novelty: 4/5. Negative temporal references combined with identity guidance address a concrete cross-video identity-transfer problem.
• Experimental Thoroughness: 4/5. Same-backbone controls, full component ablations, and two human studies are valuable, but speaker coverage and interaction samples remain limited.
• Writing Quality: 4/5. Task boundaries and ablation logic are clear, although minor metric discrepancies and corrupted extracted equations require careful reading.
• Value: 4/5. Offers a transferable recipe for lightweight identity adaptation of a strong audio-video backbone, but does not yet cover multi-person or cross-lingual generation.