PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion¶

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Code: https://github.com/showlab/PAI-Studio
Area: Others (Video Generation / Video Background Replacement)
Keywords: camera-aware motion, background replacement, foreground relighting, bidirectional attention, temporal positional encoding cloning

TL;DR¶

PAI-Studio jointly conditions video generation on a foreground video and 1–3 background images, combining real cinematographic data, temporal positional encoding cloning, and bidirectional attention to learn background motion, foreground preservation, and relighting, achieving MSE 0.00645, SSIM 0.789, and LPIPS 0.197 under multi-reference Cine-Restore evaluation.

Background & Motivation¶

Traditional green-screen workflows extract a foreground subject and composite it onto a prepared background; this is relatively straightforward with a fixed camera. With camera pushes, pans, or tracking shots, however, the background must evolve with the viewpoint rather than remain a still image behind the actor. Reference images specify the target scene's appearance, whereas the foreground video implicitly contains subject and camera motion, so both must determine background dynamics. Existing generative editors may also change the subject's identity or retain the original foreground lighting, visibly separating the subject from the new environment.

Motion is only part of the problem: practical segmentation produces flickering contours, small holes, and missing regions, which pixel-preserving compositing carries into the result. Allowing foreground regeneration helps repair boundaries and adjust illumination, but introduces risks of changes to faces, clothing textures, and body structure. The task therefore extends beyond background inpainting to joint modeling of motion, illumination, and boundaries throughout a video under foreground identity constraints. Large camera movements also reveal background content absent from the first frame, making a single reference insufficient to describe later views.

The authors use real films and online videos to supervise the coupling between foreground motion and background evolution, then align multiple background references with designated times. Training deliberately perturbs foreground illumination so that copying the input lighting cannot solve the reconstruction task. Core Idea: let one video generator jointly observe foreground dynamics, background references assigned to particular times, and motion semantics to reconstruct the whole scene under temporal alignment, rather than generate backgrounds independently and paste the subject onto them.

Method¶

Overall Architecture¶

Inputs are a foreground video, 1–3 background reference images, and optional scene text or structured motion descriptions; the output is a complete video with a replaced background and harmonized lighting. The training target is an original real video, while the conditioning foreground is segmented and photometrically perturbed, and background references are obtained by removing the subject and inpainting missing areas. The model uses a WAN 2.2-5B video Diffusion Transformer: a pretrained video VAE encodes images and videos, while a text encoder supplies text conditioning tokens. Temporal positional encoding cloning aligns background references and foreground frames with output times, and bidirectional joint attention integrates these conditions at each denoising step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real films and online videos"] --> B["Scene-restoration data construction"]
    B --> C["Structured motion guidance"]
    I["Inference inputs: foreground video<br/>background references and optional text"] --> C
    C --> D["Temporal positional encoding cloning"]
    D --> E["Bidirectional joint attention"]
    B -->|Original-video supervision, training only| E
    N["Noisy output video latents"] --> E
    E --> F["Iterative denoising and VAE decoding<br/>Complete replacement video"]

Here, camera awareness does not mean supplying an exact 3D camera trajectory, and the pipeline has no explicit depth reconstruction or camera calibration module. It primarily emerges from real-video motion statistics, textual camera descriptions, and temporal correspondence between conditioning frames and output frames. Optical flow filters training clips, and CoTracker3 evaluates motion; neither should be described as an inference-time camera controller.

Key Designs¶

1. Scene-restoration data construction: turn background replacement into supervised reconstruction

CineStudio contains approximately 30K clips from films and high-quality online videos, with each original video serving as a target that already has coherent foreground, background, and illumination. Shot segmentation is followed by optical-flow-based filtering to discard weak-motion clips and retain clear camera or background dynamics. Qwen3-VL recognizes the dominant foreground concept, supplies a short tag to SAM3 for video segmentation and tracking, and the extracted foreground is placed on a green screen. These practical segmentation outputs may retain holes, flickering, or missing regions, encouraging the model to use full spatiotemporal context to recover plausible imagery. This does not introduce a dedicated boundary-repair network or assume that every training mask is perfect.

The pipeline selects 1–3 frames from locations such as the beginning, middle, and end, removes the subject, and uses Gemini-3 Pro Image to inpaint occluded regions into clean background references. Each sample consequently provides foreground conditions, background appearance conditions, and complete video supervision without expensive paired recordings of real background replacement. However, references and targets originate from the same scene, so training remains grounded in scene restoration; cross-environment generalization requires separate evaluation. To prevent foreground illumination copying, the authors apply HSV-channel jittering, contrast adjustments, color-temperature shifts, and gamma correction. The mechanism in source Eq. (7) can be written clearly as:

\[ \widetilde{V}^{fg}=\mathcal{R}(V^{fg}). \]

The input foreground changes, but the target remains the original video, making background context necessary for recovering appropriate illumination. This is relighting learned through photometric data augmentation, not explicit estimation of light sources, reflectance, or surface normals. The original-video target still constrains texture and identity, requiring the model to distinguish subject content to preserve from illumination appearance that may change.

2. Structured motion guidance: distinguish camera semantics from subject actions

Gemini-3 provides structured JSON describing foreground content, background content, foreground motion, background motion, camera-motion analysis, camera angle, and focal length. A concise caption integrates these fields into natural language; Figure 3 labels the structured-annotation stage as Gemini-3 Flash. The fields help distinguish subject motion from camera motion while supplying scene and illumination context. For example, a tracked subject may remain centered while the background moves horizontally, so describing the subject's action alone does not specify the latter.

Structured annotations support dataset curation and can also be extracted from the input foreground video at inference time as high-level guidance. They are not mandatory manually specified per-frame trajectories, nor are their focal-length descriptions guaranteed calibrated measurements. Compared with short prompts, richer fields supply additional lighting and scene cues, but the ablation does not establish gains on every motion metric. Motion consistency is slightly higher without structured annotations in Table 4, so their role is better understood as multidimensional conditioning rather than a component that invariably improves one motion score.

3. Temporal positional encoding cloning: assign each background reference an explicit output time

Simply concatenating several background images tells the model which scene views exist, but not necessarily which view belongs at the beginning or end. The authors assign reference image \(k\) a target time \(t_k\) and copy the output video's temporal positional encoding at that time. Corresponding foreground and denoising frames at time \(t\) also receive the same temporal encoding. Source Eqs. (5)–(6) express this alignment as:

\[ \mathbf{p}(Z^{bg}_k)\leftarrow\mathbf{p}(t_k),\quad k=1,\ldots,K, \qquad \mathbf{p}(Z^{fg}_t)\equiv\mathbf{p}(Z_s(t)),\quad t=1,\ldots,T. \]

\(Z^{bg}_k\) denotes background reference tokens, \(Z^{fg}_t\) foreground-frame tokens, and \(Z_s(t)\) output tokens at time \(t\) during denoising step \(s\). What is cloned is the temporal positional encoding, not reference-image pixels or a known optical-flow field. Beginning, middle, and ending backgrounds therefore become temporally labeled conditioning anchors, with the model generating continuous changes between them. This is especially useful when camera motion substantially changes visible background content, but remains generative conditioning rather than a pixelwise hard constraint.

4. Bidirectional joint attention: reason jointly over foreground, reference backgrounds, and the emerging output

At every denoising step, the model concatenates text tokens, foreground-video tokens, background-reference tokens, and noisy output tokens into one sequence. A single-stream Transformer applies full bidirectional self-attention to that sequence instead of injecting conditions through separate control branches in succession. Source Eq. (1) organizes the sequence as:

\[ X=[Z^{txt}\Vert Z^{fg}\Vert Z^{bg}_1\Vert\cdots\Vert Z^{bg}_K\Vert Z_s]. \]

Bidirectionality allows information exchange across time and modalities rather than restricting information to a one-way foreground-to-background path. Foreground frames provide subject structure and motion, background references supply environmental appearance, and current output latents represent the emerging complete scene. The model can adjust subject illumination, complete boundaries, and generate camera-consistent backgrounds within the same generation process. The video VAE finally decodes the result, meaning the output foreground is generated rather than an immutable copy of the input pixels. This explains both the ability to repair incomplete boundaries and the need to evaluate identity fidelity separately instead of assuming that compositing guarantees it.

A Worked Example¶

The following is an explanatory scenario, not an additional experiment: a green-screen tracking shot of a person is paired with three target-scene references for the beginning, middle, and end. Structured descriptions identify the subject's action, tracking direction, and scene semantics so that background motion is not confused with the subject's own action. Each background reference is assigned its output time, while foreground frames are aligned frame by frame with the output. During denoising, joint attention combines subject dynamics and time-specific scene content to generate a continuously evolving background, not simple transitions between three still images. If the target environment is warm-toned, the model must also adjust the subject's color and shading, a behavior learned by perturbing foregrounds while reconstructing original clips during training. If segmentation misses a sleeve edge in some frames, neighboring frames may support plausible completion, but the paper does not guarantee recovery from arbitrarily large missing regions.

Loss & Training¶

The paper uses WAN 2.2-5B with rank-80 LoRA fine-tuning and AdamW at a learning rate of \(5\times10^{-5}\). Training runs in BF16 for approximately 6,000 steps on 8 H200 GPUs, with a batch size of 1 per GPU. The conditioning foreground is perturbed while the target remains the original complete video; the paper does not specify a new standalone loss formula, so identity or camera-trajectory losses should not be invented. Generation uses 97 frames at 24 FPS, approximately 4 seconds, at \(1920\times1056\) resolution. The authors call this 1080P, but it is not the conventional \(1920\times1080\) size; reproduction should follow the reported dimensions. The main paper does not fully specify the sampling algorithm, sampling steps, inference latency, or all illumination-augmentation ranges, and does not establish real-time generation.

Key Experimental Results¶

Main Results¶

Cine-Benchmark separates original-scene restoration from novel-background generation so that pixel reconstruction is not mistaken for cross-scene generalization. Cine-Restore contains 150 CineStudio videos disjoint from training, using background references extracted and inpainted from their corresponding original clips. Cine-NBG contains 50 videos from DAVIS 2017 and IVEBench, with Gemini-modified backgrounds introducing new styles and layouts and no corresponding ground-truth videos. The open-source baseline is VACE-1.3B fine-tuned on the same data and resolution; commercial systems use default API configurations, so model sizes and actual system budgets are not fully matched.

The following selection from source Table 2 reports Cine-Restore results; Single / Multi denote single-reference and multi-reference protocols. MSE, SSIM, and LPIPS measure reconstruction similarity to the ground-truth video, while motion consistency uses CoTracker3 trajectory comparisons. Illumination harmony is a Gemini perceptual score on a 1–10 scale; the main paper does not supply the full trajectory-score formula or evaluation prompts.

Method and condition	MSE ↓	SSIM ↑	LPIPS ↓	Motion Cons. ↑	Illum. Harmony ↑
VACE, Single	0.0270	0.595	0.393	0.792	7.67
Kling O1, Single	0.0282	0.646	0.402	0.706	7.62
PAI-Studio, Single	0.0164	0.717	0.303	0.872	9.11
Kling O1, Multi	0.0190	0.682	0.341	0.721	7.44
Kling 3.0 Omni, Multi	0.0210	0.622	0.370	0.715	7.44
PAI-Studio, Multi	0.00645	0.789	0.197	0.894	9.61

Multiple references increase both visible scene information and temporal anchors, so Single-to-Multi gains cannot be attributed entirely to positional encoding. Under the same Multi setting, PAI-Studio achieves LPIPS 0.197 versus Kling O1's 0.341 and motion consistency 0.894 versus 0.721. The next selection is from source Table 3 on Cine-NBG: all entries are Gemini scores on a 1–10 scale and must not be mixed with Table 2's trajectory scores.

Method	Motion Cons. ↑	Edge Quality ↑	Fg. Pres. ↑	FG-BG Fusion ↑	Illum. Harmony ↑
VACE	6.92	7.82	2.44	7.04	8.38
Kling O1	7.12	8.04	6.48	7.36	7.72
Switchx	7.12	7.40	8.88	6.16	7.88
PAI-Studio	8.16	8.86	9.58	8.40	9.22

Foreground preservation assesses subject identity, edge quality assesses transitions around segmentation defects, and fusion assesses overall geometric, photometric, and temporal integration. These perceptual dimensions complement reconstruction metrics but are neither identity-verification accuracy nor physical illumination error.

Ablation Study¶

The following selection from source Table 4 uses the Cine-Restore Multi protocol for every row, allowing component comparisons under the same reference-image setting.

Config	MSE ↓	SSIM ↑	LPIPS ↓	Motion Cons. ↑	FG-BG Fusion ↑	Illum. Harmony ↑
w/o Position Encoding Control	0.0160	0.681	0.287	0.843	8.41	8.88
w/o Illumination Adaptation	0.0130	0.681	0.279	0.879	7.24	7.31
w/o Structured Annotation	0.00695	0.787	0.198	0.902	9.21	9.57
Full Model	0.00645	0.789	0.197	0.894	9.33	9.61

Key Findings¶

Removing positional encoding control lowers motion consistency from 0.894 to 0.843 and increases LPIPS from 0.197 to 0.287, supporting temporal anchoring for both motion and scene fidelity.
Removing illumination adaptation lowers illumination harmony from 9.61 to 7.31 and fusion from 9.33 to 7.24, the largest deterioration in these two columns among the listed configurations.
Removing structured annotations increases motion consistency from 0.894 to 0.902 but worsens MSE, fusion, and illumination; the full model is not best on every ablation metric.
Source Section 4.5 and Figure 7 report 25 participants with relevant experience, with PAI-Studio receiving 29.7%–35.6% of best-result votes across five criteria.
The authors report significance at \(p<0.01\) across criteria; best-result vote shares are not pairwise win rates, and the cached main paper does not provide full paired statistics.

Highlights & Insights¶

Supervision directly targets practical mismatch: preserving the original target while perturbing foreground lighting makes environment-dependent relighting necessary for solving the task.
The value of multiple references lies not only in their number but also in their assigned output times. Temporal positional encoding establishes inexpensive correspondence without a separate camera-trajectory predictor.
Whole-video generation incorporates boundary repair and illumination harmonization into one process. It relaxes hard compositing's dependence on perfect masks but makes identity preservation an empirical capability rather than a guarantee.
Separating restoration and novel-background evaluation improves interpretability. The former tests known-scene reconstruction and the latter tests cross-background generalization; their metrics should not be merged into a single ranking.

Limitations & Future Work¶

The following are reading-based assessments of the main paper's setup; the paper has no dedicated, systematic failure-case section. - Approximately 4 seconds across 97 frames does not establish long-shot stability. Longer durations, shot changes, and multi-subject occlusions require dedicated testing. - Without explicit 3D camera control or physical illumination modeling, camera awareness does not imply exact trajectory execution, and relighting does not imply editable lighting parameters. - Cine-NBG has only 50 videos and relies on Gemini scoring; both data construction and evaluation use the Gemini family, warranting independent judges and complementary metrics to check potential evaluation preferences. - VACE-1.3B and WAN 2.2-5B differ in capacity, and commercial APIs differ in versions and configurations, so gains cannot all be assigned to the proposed control mechanism alone. - The main paper lacks systematic mask-corruption severity curves, identity failure rates, and runtime measurements, preventing conclusions about arbitrary missing foreground regions or real-time production readiness. - Controlled camera-trajectory tests, long-duration benchmarks, and stricter subject-detail evaluation could distinguish visually plausible generation from geometric and identity accuracy.

vs VACE: both address video creation and editing; PAI-Studio focuses on joint foreground-video and multi-background-image conditioning, while the same-data baseline still differs in model capacity.
vs traditional video matting: traditional pipelines deliver an accurate foreground layer and rely on prepared backgrounds; PAI-Studio generates the full scene to address background motion and lighting mismatch, sacrificing guaranteed foreground pixel preservation.
vs Kling O1 and Switchx: commercial methods provide application-oriented references; Switchx scores 8.88 on foreground preservation in Table 3 versus 9.58 for PAI-Studio, but this is a model-judged score on a particular benchmark.
Transferable insight: when reference images correspond to different video times, establish temporal semantics before asking a generator to infer transitions; transfer to other editing tasks still requires sufficient motion supervision in the data.

Rating¶

Novelty: 4/5. The combination of task constraints, temporal anchoring, and illumination perturbation is clear; the main contribution is task and training design rather than a new attention operator.
Experimental Thoroughness: 4/5. Restoration, novel backgrounds, component ablations, and a user study are covered, but sample size, judge independence, and efficiency reporting remain limited.
Writing Quality: 4/5. The methodological thread and tables are clear, while some evaluation implementation and statistical details require supplementary material.
Value: 4/5. The work offers reusable ideas for cinematic background replacement, but the main paper does not establish long-shot production reliability.