In-Context Sync-LoRA for Portrait Video Editing¶

Conference: ECCV2026
Paper: ECCV Paper
Project: Sync-LoRA
Area: Video Generation
Keywords: portrait video editing, first-frame edit propagation, in-context learning, synchronization filtering, LoRA

TL;DR¶

Sync-LoRA trains an in-context LoRA for LTX-Video on highly synchronized synthetic video pairs, letting an edited first frame specify appearance and a clean source video specify motion, achieving a useful balance between edit fidelity, frame-level synchronization, and identity preservation rather than leading every synchronization metric.

Background & Motivation¶

Portrait video editing must do more than keep an added hat stable across consecutive frames: it must preserve when the original performance happens. A blink, gaze shift, or mouth shape associated with a syllable can look natural in the output yet alter the performance if it occurs earlier or later than in the source. Temporal coherence therefore only ensures that the output does not flicker internally; it does not ensure frame-by-frame correspondence with the input. Text-guided editing can preserve much of the original footage without implementing the requested change, while first-frame-guided methods can change appearance but generate a different subsequent motion trajectory. Portrait animation based on keypoints, 3D morphable face models, or geometric warping provides control, but its representations and synthesis mechanisms can constrain open-ended editing and occlusion handling.

Delegating the first frame to a mature image editor resolves the desired appearance without automatically resolving motion in every subsequent frame. Given only an edited image, an image-to-video model can produce plausible motion but lacks sufficient evidence to reproduce the source video's fine timing. If paired training subjects merely speak approximately together, the model learns similar motion rather than strict temporal correspondence. The paper consequently focuses on synchronized supervision and the contextual representation of the source video instead of adding an explicit motion-correspondence network. This explains its emphasis on a small, high-quality paired dataset: quantity cannot compensate for incorrect temporal supervision.

Static IC-LoRA already shows that diffusion transformers can learn cross-image relationships from structured panels. The challenge here is to extend appearance correspondence to correspondence at every moment, without letting appearance edits disrupt the temporal reference. Core Idea: train an in-context LoRA on precisely synchronized videos with different appearances while always exposing the clean source video, so the model propagates the first-frame edit instead of inventing the source motion again.

Method¶

Overall Architecture¶

Inference takes a source video, its edited first frame, and text describing the target content or editing relationship, and returns the edited sequence. The method has an offline data-and-training path and an editing path using the trained LoRA; these are not stages of a single inference run. Offline processing performs Paired Video Generation, applies Four-Channel Synchronization Filtering, and trains Clean-Source In-Context Denoising. Online editing runs neither the data generator nor the synchronization filter: a new source video and editing conditions go directly to the adapted LTX-Video. Wan2.1 generates training videos, whereas LTX-Video performs editing; the former is not a second editing model invoked during inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Prompts["Subject descriptions<br/>and edit instructions"] --> Generation["Paired Video Generation"]
    Generation --> Filtering["Four-Channel<br/>Synchronization Filtering"]
    Filtering -.->|Offline paired supervision| Context["Clean-Source<br/>In-Context Denoising"]
    Inputs["Source video, edited first frame,<br/>and text"] --> Context
    Context --> Output["Synchronized edited video"]

Key Designs¶

1. Paired Video Generation: create a shared performance with different appearances

A VLM produces subject descriptions and editing instructions, QwenImage generates a portrait, and QwenImageEdit-2509 edits it. The images are combined into a left-right panel, which Wan2.1 converts jointly into a side-by-side talking-head video rather than sampling two independent clips and forcing them into a pair. This exploits the video diffusion model's existing in-context generation ability to encourage shared motion patterns across the two views. The authors also LoRA-fine-tune the data-generation model on a small curated set of synchronized examples to improve shared-motion generation. Each paired video uses 30 denoising steps during data generation; this is a Wan2.1 setting, not the Sync-LoRA inference step count. The two views become source and target supervision, ideally differing in edited appearance rather than action timing.

Joint generation does not guarantee frame-level agreement: the subjects can open their mouths at different rates or blink and turn their heads at different moments. The authors therefore generate over 20,000 candidate pairs but use only 512 pairs for final training; the candidate count is not the training-set size. Edited pairs and identical, unmodified pairs appear in a 3:1 ratio, with the latter providing a stabilizing signal for retaining original content. More identical pairs are not necessarily better: a dataset requiring no appearance changes encourages the model to preserve motion while ignoring edits. Using only edited pairs also weakens this stabilizing constraint, and the ablation shows reduced motion synchronization. The value lies not merely in synthetic data, but in supervision where appearance differences coexist with temporal agreement.

2. Four-Channel Synchronization Filtering: verify that matching actions occur at matching moments

MediaPipe extracts facial and upper-body landmarks, which are converted into comparable temporal signals. Speech uses mouth aspect ratio as a proxy for articulation, while Blink uses the Eye Aspect Ratio, or EAR, to describe eyelid changes. Gaze uses normalized two-dimensional iris trajectories, and Pose uses six upper-body angle or relative-height signals involving shoulders, torso, elbows, and wrists. These channels cover movements with different speeds and amplitudes, preventing similar mouth motion from hiding asynchronous gaze. Missing detections are interpolated, signals are smoothed with a Savitzky-Golay filter, and z-normalization is applied. Pairs with insufficient detection coverage are discarded rather than treating extensive interpolation as reliable observation.

Corresponding signals are correlated at zero temporal lag, directly matching the goal of frame-level synchronization. Searching for the best time offset could incorrectly accept a delayed blink as a high-quality match and retain timing errors in the training set. Normalization helps compare movement rhythms across appearances and amplitudes, but correlation does not directly guarantee identical absolute geometry. Writing the four channel scores as \(r_{\mathrm{speech}}\), \(r_{\mathrm{gaze}}\), \(r_{\mathrm{blink}}\), and \(r_{\mathrm{pose}}\), the weights stated in the paper can be expressed as:

\[ S=0.40r_{\mathrm{speech}}+0.30r_{\mathrm{gaze}}+0.15r_{\mathrm{blink}}+0.15r_{\mathrm{pose}}. \]

This equation directly expresses the reported weights without inventing the internal aggregation of multiple signals within each channel. Retaining highly ranked pairs supplies the model with training targets that have accurate temporal correspondence. The same four correlations are used for final evaluation, but they do not enter the training loss or provide corrective feedback during inference. Removing a channel in the ablation means setting its filtering weight to zero and changing the selected data, not disabling a facial component inside the model. Consequently, worse mouth motion after removing blink filtering does not demonstrate a blink network controlling the mouth; lower-quality pairs can jointly corrupt several motion channels.

3. Clean-Source In-Context Denoising: retain the original performance as a reference at every step

LTX-Video compresses video into spatiotemporal latents and processes them with a DiT using full three-dimensional attention. Source and target latent sequences are concatenated along the frame dimension inside the same transformer, with text and first-frame conditions guiding generation. Concatenation provides a common attention space; it does not mean interpreting the source and target as consecutive segments of one performance. Both streams share the same spatiotemporal coordinates and 3D RoPE, giving corresponding spatial and temporal positions a common positional reference. Coordinates are scaled by VAE compression factors and normalized by frame rate; these describe video geometry rather than diffusion noise levels. A separate signal distinguishes the streams: the per-token denoising timestep.

Source tokens always use \(t=0\), while target tokens awaiting denoising use the current \(t>0\). These timesteps generate AdaLN scale and shift parameters, allowing the model to interpret source tokens as clean conditions and target tokens at matching positions as content to update. Only the target branch receives noise and prediction supervision during training; source-video latents remain unchanged. It is the input source stream that stays fixed, not the target's ability to access source information through shared attention. Under this supervision, LoRA learns to retrieve motion from the source video and desired appearance from the edited first frame. Inference needs no explicit keypoint-correspondence estimation, but training-data selection does depend on keypoints, so the complete method should not be described as keypoint-free.

Inference initializes the target from Gaussian noise and progressively updates its latents through a flow-matching Euler loop. Every step retains access to the complete clean source stream, avoiding the gradual loss of a one-time motion hint during generation. The first frame specifies the new appearance, text supplements the semantic relationship, and the source video provides the timing reference for motion. A trained general-purpose LoRA handles unseen identities and multiple appearance edits without training an adapter for every test subject. However, Section 4.4 trains an expression-specific Sync-LoRA separately, so that demonstration does not establish unrestricted expression editing with the same general-purpose module.

A Worked Example¶

Consider the paper's Figure 4 example of adding a blue bandana: first edit the source video's first frame so the bandana placement and subject geometry remain aligned with the original image. This image specifies the bandana's appearance, not when the subject should speak or shift their gaze. The encoded source video stays clean while the target starts from noise; every denoising step reads motion context from the corresponding source moments. The output should preserve the timing of the original head turns, speech, and blinks while letting the bandana move naturally with the person. A stable bandana with delayed mouth motion is a synchronization failure; perfectly reproduced motion with a missing bandana is an editing failure. This is why the paper evaluates motion, edit direction, and identity together instead of reporting only a correlation score.

Loss & Training¶

The authors adapt LTX-Video with rank 128 LoRA using LTX-Video-Trainer, without replacing the base architecture. Training retains rectified-flow velocity prediction, conditioned on text, the edited first frame, and the source video, with loss applied only to the edited target branch. The loss equation on page 9 is visibly corrupted in the available text, and its interpolation endpoints and velocity-target notation cannot be reliably reconciled; this note therefore does not reconstruct it as the authors' exact equation. What is clear is that the objective is not a weighted sum of four-channel correlations: data filtering and velocity prediction are separate operations. The cache does not contain the referenced supplementary pseudocode, and the learning rate, training steps, and final inference step count cannot be reliably recovered from the main text. Expression-specific training uses LivePortrait to synthesize videos with different expressions from a neutral reference frame, then learns to preserve underlying articulation while changing expression; the presented evidence is primarily qualitative.

Key Experimental Results¶

Main Results¶

The benchmark contains 166 portrait videos from what the main text calls CelebV, CelebV-HQ, TalkVid, and high-quality YouTube content, covering object insertion, background replacement, colorization, and appearance modification. Source selection requires at least 5 seconds of sharp, well-lit footage and excludes subtitles, letterboxing, cropping, and jump cuts; actual training and evaluation clips contain 81 frames at 20 FPS, approximately 4 seconds. Image-conditioned methods use the same edited first frame; LucyEdit accepts only text, making image-based Directional CLIP inapplicable. The table below selects results from Table 1 on page 10; higher is better for every metric, and the three VACE rows represent different external control conditions.

Method / Condition	Speech Corr.	Gaze Corr.	Blink Corr.	Pose Corr.	Directional CLIP (text-dual)	ArcFace Sim.
VACE / Pose	0.37	0.63	0.34	0.45	0.20	0.73
VACE / Canny	0.11	0.63	0.24	0.39	0.20	0.71
VACE / Depth	0.39	0.69	0.36	0.51	0.16	0.72
LucyEdit	0.80	0.82	0.70	0.64	0.11	0.69
FlowEdit	0.50	0.56	0.33	0.38	0.18	0.68
AnyV2V	0.70	0.71	0.48	0.51	0.17	0.63
Sync-LoRA	0.72	0.75	0.55	0.55	0.21	0.75

Image-based Directional CLIP compares each frame's edit direction with the first-frame edit direction, while the text-dual variant compares visual changes with the edit direction described by text. CLIP-Text Alignment measures generated-frame agreement with the target description without comparing source frames; Ours scores 0.33 and does not uniquely lead every baseline. ArcFace averages cosine similarity between generated-frame face embeddings and the edited first-frame embedding, so it is not direct proof of absolute identity agreement with the original person.

Ablation Study¶

The following selection comes from Table 2 on page 13 under the same evaluation framework; removing a channel only changes training-data filtering. Directional CLIP (image) corresponds to the unqualified Directional CLIP row in Table 2.

Training Config	Speech Corr.	Gaze Corr.	Directional CLIP (image)	Directional CLIP (text-dual)	ArcFace Sim.
Without Speech filtering	0.53	0.68	0.55	0.20	0.72
Without Gaze filtering	0.56	0.67	0.57	0.20	0.72
Without Blink filtering	0.55	0.70	0.56	0.20	0.72
Without Pose filtering	0.55	0.67	0.56	0.20	0.72
Only Edit	0.58	0.68	0.57	0.21	0.73
Only ID	0.80	0.83	0.05	0.01	0.70
Random	0.56	0.68	0.55	0.20	0.72
Full model	0.72	0.75	0.57	0.21	0.75

Key Findings¶

Relative to LucyEdit, Sync-LoRA lowers Speech Corr. from 0.80 to 0.72 but raises the text-dual directional score from 0.11 to 0.21; the finding is an improved editing-synchronization trade-off, not universal synchronization leadership.
Removing Speech filtering reduces Speech Corr. from 0.72 to 0.53, an absolute decrease of 0.19, supporting the importance of synchronized supervision quality.
Only ID reaches 0.80 Speech Corr. but only 0.05 image-based Directional CLIP; high motion similarity can arise from barely editing the video.
The user study includes 23 participants, each judging 12 video pairs with 3 per baseline, yielding 69 judgments per competitor; exact preference rates are in supplementary material absent from the cache and cannot be reconstructed from the main text.

Highlights & Insights¶

Training-data errors can determine synchronization quality more strongly than architecture changes. Once input-target timing is correct, ordinary low-rank adaptation can exploit the base video model's existing capabilities.
The clean source stream preserves a motion reference at every denoising step. This differs fundamentally from generating plausible future actions from a first-frame condition alone.
Only ID provides an informative counterexample. It shows why editing evaluation must not mistake leaving the video unchanged for high-quality video editing.

Limitations & Future Work¶

The authors acknowledge reduced detail stability under fast or large motion, and local artifacts or temporal drift when the edited first frame is geometrically misaligned with the source.
Current evidence concerns 81-frame clips; overlapping sliding windows are supported, but long-form degradation has not been studied specifically.
Filtering and evaluation share the four-channel metrics, potentially favoring movements captured by landmarks; this is a reader assessment of evaluation coverage, not established data leakage.
Curated clear footage and a small synthetic training set do not establish robustness to complex shots, multi-person interactions, or every identity distribution.
Audio conditioning and stress tests for long sequences and geometric misalignment follow directly from the current limitations; they are not capabilities already demonstrated here.

IC-LoRA / PoseGen: the former learns static-panel relationships and the latter emphasizes pose-controllable video; this paper supervises precise temporal correspondence and first-frame edit propagation.
AnyV2V / VACE / FlowEdit: these are editing baselines actually evaluated in the paper; advantages should be judged under the shared experimental setting, not extrapolated to every task from their general method descriptions.
LivePortrait: warping-based reenactment is both related work and a source of expression-specific training data; the paper discusses weaknesses under large head rotations and occlusion without providing an independent quantitative comparison table in the main text.
Transferable insight: zero-lag filtering for other measurable motion attributes, coupled with a clean conditioning stream, is worth testing for high-fidelity editing; cross-domain effectiveness still requires new experiments.

Rating¶

Novelty: 4/5, primarily synchronized supervision construction and video in-context training rather than a new base network.
Experimental Thoroughness: 4/5, multidimensional comparisons and filtering/data-composition ablations, but limited long-video and out-of-distribution evaluation.
Writing Quality: 4/5, a clear task and data rationale; the cached loss equation is corrupted, so precise reproduction still requires the original typesetting and supplementary material.
Value: 4/5, a practical data-first approach to portrait editing that preserves the original performance.