Layer-Aware Video Composition via Split-then-Merge¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: layered video composition, transformation-aware augmentation, identity preservation, affordance, self-composition supervision

TL;DR¶

Split-then-Merge (StM) learns reconstruction from automatically decomposed unlabeled videos, using perturbed foreground conditions and region-weighted supervision to integrate existing dynamic subjects into new backgrounds, outperforming comparison methods on foreground identity, background identity, action, and background motion metrics in cross-video DAVIS composition tests.

Background & Motivation¶

Video composition involves more than extracting a subject and pasting it into another clip: the original action, background camera motion, grounding, and local lighting must coexist. Traditional color transfer, harmonization, and gradient-domain composition primarily handle pixel transitions, but do not decide that a swan belongs in water instead of on the shore. Generative methods can redraw contact shadows and adapt orientation and illumination, yet a single conditioning image easily loses the original motion. For example, composing the first frame with Qwen-Edit and then using an image-to-video (I2V) model supplies static appearance rather than the subject's complete motion sequence.

Providing a video generator with two videos does not automatically solve composition either. When training foreground and background layers come from the same source clip, the original position reveals the answer: the model can reconstruct by copying the foreground without learning how it should fit a new scene. Conversely, redrawing the subject freely for visual harmony can change its identity, as when a black swan becomes white in the paper's Figure 2. The task therefore requires retaining motion and identity while permitting changes to position, color, and contact relationships; neither pixel fidelity alone nor visual appeal alone is sufficient.

StM addresses the source of supervision and the difficulty of training, instead of writing placement rules for each object category. Real videos already contain plausible subject-environment relationships, and automatically splitting them into layers makes the original clip a composition target without manually prepared paired composites. However, that supervision becomes useful only when copying shortcuts are disrupted, so the authors perturb the foreground condition and ask the model to recover appropriate spatial and lighting relationships. Core Idea: use reconstruction of decomposed real videos as composition supervision, force placement and harmonization through foreground transformations, and apply region-weighted identity preservation to prevent the subject from becoming a different object while adapting to a new background.

Method¶

Overall Architecture¶

The inputs are a foreground video with a binary subject mask, a separate dynamic background video, and text describing the target scene; the output is a newly generated composite video. Training and testing differ in pairing: training layers come from the same clip and reconstruct it, whereas testing layers come from different clips and form a previously unseen combination. The model should retain foreground semantic action and identity as well as background scene and camera motion, but does not promise pixel-exact copying of every element.

The training pipeline comprises Automatic Decomposition, Transformation-Aware Augmentation, Multi-Layer Conditional Fusion, and Identity-Preservation Supervision, in that order. Decomposition supplies the data; augmentation changes only the foreground condition; fusion connects the video layers to a pretrained generator; identity supervision allocates reconstruction error between foreground and background. The main chain below depicts training data and supervision, with dashed edges identifying the original target and the partition mask, rather than ground-truth requirements at test time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Original["Unlabeled original video"] --> Split["Automatic Decomposition"]
    Split -->|Foreground video| Augment["Transformation-Aware<br/>Augmentation"]
    Augment --> Fusion["Multi-Layer<br/>Conditional Fusion"]
    Split -->|Background and text| Fusion
    Original -->|Noisy target latent| Fusion
    Fusion --> Denoise["DiT predicts composite latent"]
    Denoise --> Identity["Identity-Preservation<br/>Supervision"]
    Split -.->|Foreground mask| Identity
    Original -.->|Reconstruction target| Identity

Inference omits training augmentation and identity supervision; sampled noise replaces the noisy original-video latent, denoising uses foreground, background, and text conditions, and the space-time VAE decodes the result. Thus, the original-video target in the diagram is not an inference input, and the identity loss is not an additional test-time recognizer or optimization procedure. The learned Composer adapts the relative placement of the input layers; manually specified trajectories and a physics simulator are not required components.

Key Designs¶

1. Automatic Decomposition: turn real videos into reconstructable layered samples

The Decomposer produces a caption, a dynamic foreground, its mask, and an inpainted background for each video. The implementation specifies InternVL for captioning, Segment-Any-Motion for extracting the primary moving subject, and MiniMax Remover for filling the hole left after subject removal. The subject mask separates the layers, while background inpainting avoids supplying a background with a hole as if it were a normal input. The goal is not a manually refined ground-truth background, but enough automatically produced examples with usable subject and scene information. The original clip remains the target, so the model learns to reorganize these separated, potentially imperfect conditions into a real scene.

The authors source animal videos from Animal Kingdom, human videos from Panda-70M, and general objects from Youtube-VOS and LVOS, producing 50,000 clips in StM-50K. This avoids separately recording empty and occupied versions of each scene or manually annotating every target composite. Nevertheless, no manual annotation does not mean no supervision: the original video provides an explicit reconstruction target, while pretrained segmentation, captioning, and inpainting models supply automatic conditions. Decomposition quality still affects the data through missing subject content, inaccurate boundaries, and background inpainting artifacts; the later robustness experiment targets mask errors. A distribution gap remains between training self-composition and test-time cross-video composition, motivating the next design to prevent memorization of existing layer alignment.

2. Transformation-Aware Augmentation: disturb only the foreground condition while retaining the correct target

If foreground orientation, size, position, and color already match the original clip, copying is the easiest reconstruction strategy. The authors apply random horizontal flipping, cropping and resizing, and color perturbation only to the conditioning foreground, leaving the background and original-video target unchanged. Transformations are temporally consistent rather than sampled independently per frame, which would introduce artificial flicker or motion changes. The model must use background and text information to restore an appropriate relationship for the perturbed subject while retaining its action content. Inverting transformations describes this training requirement; it does not imply that the system explicitly estimates a transformation matrix and analytically reverses it.

Horizontal flipping and random resized cropping each have probability 0.7; the crop scale range is [0.5, 2.0], with 90% of the foreground preserved. Color jitter has probability 0.2 and affects only the masked foreground, with brightness, contrast, saturation, and hue perturbation ranges reported as [0, 0.2]. Spatial augmentation breaks reliance on fixed layouts, while color augmentation prevents dependence on a foreground whose tones already match the background. Affordance here denotes learned relationships about where a subject belongs and how it should contact the scene, rather than an explicit affordance classification label. For example, an aquatic subject should fit the water surface, but the model does not thereby acquire a strict guarantee of physical correctness.

3. Multi-Layer Conditional Fusion: expose every latent location to both videos

The Composer adapts pretrained CogVideoX-I2V weights and uses a frozen space-time VAE to encode the original video, background, and augmented foreground. At a training denoising step, the original latent is noised, concatenated channel-wise with background and augmented foreground latents, and projected by an MLP into a visual representation. This supplies dense spatiotemporal conditions at each latent location instead of compressing the entire foreground into an identity vector or suggesting motion through the first frame alone. Because the foreground has been spatially perturbed, matching grid locations do not imply correct subject placement; the DiT must still use context to relocate and integrate it. A text encoder produces text representations, which are concatenated with visual tokens along the token dimension before the DiT predicts a composite latent for reconstructing the original.

Channel fusion and text-token concatenation serve different purposes: the former retains dense video-layer information, and the latter supplements target-scene semantics. Compared with a subject-image-only pipeline, complete video conditions expose the evolution of actions and the background's own camera and scene dynamics. Unlike LayerFlow's task of generating layers from text, the conditions here are existing user videos that should largely be retained rather than imagined from scratch. The authors state that only the projection layer introduces new parameters and report inference time identical to CogVideoX-I2V; no concrete latency or memory figures are given in the main text to quantify throughput. The model still generates new pixels in latent space rather than combining two decoded videos with fixed alpha blending.

4. Identity-Preservation Supervision: average reconstruction errors by region instead of letting background area dominate

A standard reconstruction objective treats latent locations uniformly, so errors on a small subject can be diluted by a large background. The authors use the training foreground mask to partition prediction-versus-original latent error into foreground and background terms, normalize each by its corresponding area, and combine them with weights. This makes foreground importance less directly dependent on its pixel footprint and explicitly controls the emphasis on identity preservation versus overall harmony. Despite its name, the identity-preservation loss is masked latent reconstruction supervision, not an additional face-recognition, contrastive-learning, or ViCLIP identity loss. ViCLIP is primarily used for evaluation here and should not be mistaken for a training teacher.

Augmentation and regional supervision work together: augmentation makes copying insufficient, and regional supervision limits subject-appearance damage during recomposition. The full model does not outperform every ablation on every foreground fidelity metric; the authors use this observation to caution that excessive similarity to the input may indicate a copying shortcut. The original clip and its mask are available during training for region-specific targets, whereas no target composite exists at test time, so this constraint acts through the learned parameters. Equations (1) through (4) are visibly corrupted in the extracted text; this note explains fusion and regional losses from surrounding prose and Figure 4 without reconstructing the authors' exact formulas. The implementation explicitly gives the identity-preservation weight as \(\alpha=0.5\), but this value cannot recover the damaged equation notation.

A Worked Example¶

Consider the swan composition in Figure 2: the foreground video provides the swan's appearance and continuing movement, while the background supplies the dynamic shore and water scene. Comparable training relationships originate in real clips: decomposition extracts the subject and inpainted background, and the model must reconstruct the original plausible relationship despite perturbing the subject condition. At test time, random perturbations are disabled and the two input video latents are supplied directly to the Composer. Unlike composing a first frame and extending it into a video, the full swan action sequence remains a condition; unlike copy-paste, the output may adapt placement and environmental interaction. The authors' example places the swan in water and generates waves and shadows; these are qualitative observations described in the figure and text, not independent measurements of physical accuracy. The interpretation here is that the reusable principle is preserving semantic action while adapting contact relationships, rather than guaranteeing every input pixel remains unchanged.

Loss & Training¶

The Composer is fine-tuned for 20K iterations on 16 NVIDIA H100 GPUs with batch size 64 and bf16 precision, using videos of size \(49\times480\times720\). AdamW uses \(\beta_1=0.9\), \(\beta_2=0.95\), and \(\epsilon=10^{-8}\), with weight decay \(10^{-4}\) and maximum gradient norm 1.0. A cosine learning-rate schedule uses a base learning rate of \(5\times10^{-6}\) and 1K warm-up iterations. The space-time VAE is frozen; the main text describes generator fine-tuning from pretrained weights but does not fully enumerate the freezing status of all text-encoder parameters here. Training reconstructs the original video from same-source layers; the 93 DAVIS triplets with cross-video foregrounds and backgrounds evaluate compositional generalization only.

Key Experimental Results¶

Main Results¶

The test set comprises 93 unseen foreground-background-text triplets from DAVIS, with foreground and background originating in different videos. Table 1 on page 8 defines M1/M2 as cosine similarities between ViCLIP representations of input and generated layers, measuring foreground and background identity preservation respectively. M3 is KL divergence between foreground VideoSwin action representations, measuring semantic action discrepancy; M4 is mean squared error between background optical flows, measuring camera and scene motion discrepancy. M5 is ViCLIP cosine similarity between the text and generated video; M1, M2, and M5 are multiplied by 100 and are higher-is-better, while M3 and M4 are lower-is-better. M1 through M4 require decomposing generated videos into corresponding layers, so they depend on decomposition quality and do not directly measure contact forces or physical correctness.

The following results are from Table 2 on page 10; cascaded I2V baselines share the CogVideoX-I2V backbone, inference steps, and classifier-free guidance scale with StM, but not all comparison methods receive equivalent dynamic conditions.

Method	M1 FG identity ↑	M2 BG identity ↑	M3 semantic action ↓	M4 BG motion ↓	M5 text ↑
Copy-Paste + I2V	83.08	85.02	1.61	184.59	19.21
PBE + I2V	73.52	80.19	2.47	98.08	19.41
Qwen + I2V	82.02	72.38	1.71	74.77	24.20
SkyReels	80.24	75.24	1.75	279.23	24.40
AnyV2V	77.73	56.36	1.70	154.97	24.13
LayerFlow-FG	83.02	60.21	1.47	143.47	18.74
LayerFlow-BG	50.27	91.73	2.35	24.72	19.57
StM	84.82	92.88	1.22	16.36	19.81

Full StM leads the comparison methods on M1 through M4, but its textual alignment trails SkyReels at 24.40; superiority does not extend to M5. Table 3 on page 11 also reports a user study with 50 participants on 25 randomly sampled cases and Gemini 2.5 Pro judgments on the full test set. Against Qwen + I2V, StM wins 55.10% of human overall-quality comparisons but only 46.73% of VLM overall-quality comparisons, showing that perceived appeal need not coincide with identity and motion fidelity.

Ablation Study¶

The following reproduces the component ablations in Table 2 on page 10 under the same evaluation metrics; absolute scores avoid describing differently directed metrics uniformly as degradation.

Transformation augmentation	Identity-preservation loss	M1 ↑	M2 ↑	M3 ↓	M4 ↓	M5 ↑
Yes	Yes	84.82	92.88	1.22	16.36	19.81
Yes	No	82.01	88.25	0.92	23.75	16.42
No	Yes	83.15	89.20	0.70	16.61	16.56
No	No	90.39	84.76	0.77	18.59	18.70

Removing identity supervision raises M4 from 16.36 to 23.75, a relative deterioration of 45.2% as reported in the original table; M1 and M2 also worsen, but M3 improves from 1.22 to 0.92. Removing both components instead yields M1 of 90.39, while removing augmentation alone produces the lower M3 of 0.70. The authors interpret these favorable foreground scores as copying shortcuts rather than better composition; the numbers support a fidelity-integration trade-off but do not independently prove better affordance in every case.

Table 4 on page 15 corrupts foreground masks at inference; the excerpt below uses SAM-3 for evaluation decomposition, so its absolute scores must not be mixed directly with Table 2.

Mask condition	M1 ↑	M2 ↑	M3 ↓	M4 ↓	M5 ↑
Clean input	82.22	90.60	1.06	16.88	19.81
Random pixel masking 25%	77.08	88.66	1.41	17.33	19.29
Random pixel masking 75%	73.87	88.22	1.81	27.00	18.47
Boundary erosion 25%	78.28	88.49	1.32	17.24	19.37
Boundary erosion 75%	74.06	87.51	1.63	21.20	19.03

Key Findings¶

Background motion retention is a major comparative strength: StM reaches M4 of 16.36 versus 24.72 for the strongest comparison baseline, LayerFlow-BG, in Table 2.
The ablation is a joint analysis of fidelity and adaptation, not evidence that every component improves every score monotonically; foreground metrics especially require background metrics and qualitative context.
Mask-corruption tests show generally controlled degradation, but severe random masking raises M4 to 27.00; robustness should not be interpreted as insensitivity to decomposition errors (Table 4).

Highlights & Insights¶

Real scenes can supply composition supervision. Decomposition followed by reconstruction converts labor-intensive paired editing targets into automatically constructible training data.
Augmentation deliberately makes the condition differ from the target rather than demanding transformation-invariant outputs. The model must use scene semantics to correct placement and appearance before it can learn new combinations.
Identity metrics are insufficient measures of composition quality. The ablations show that copying can improve foreground scores without ensuring correct environmental relationships.

Limitations & Future Work¶

The authors explicitly demonstrate scale ambiguity and severe-occlusion failures: a subject can be oversized relative to a street, and an occluded panda cannot be fully recovered (Figure 8, page 14).
For impossible prompts, the model may turn part of a road into water to accommodate a swimming subject (Figure 7, page 13). This demonstrates generative flexibility but also shows that background preservation is not a hard constraint.
Evaluation remains limited to 93 combinations and 25 user-study cases; recursive multi-object insertion is primarily qualitative and does not quantify long-sequence, multi-object stability.
The authors propose relative scale priors for objects and scenes; this note further suggests explicit contact constraints and editable-background regions, but these are not implemented components of StM.
The available full text contains the main paper and references, not the cited supplementary material or videos; damaged equations, supplementary decomposer comparisons, and focused affordance-evaluation details were not independently verified.

vs Qwen + I2V / SkyReels: static-conditioning pipelines can produce appealing videos but cannot fully access original foreground actions; StM prioritizes action and identity retention through dynamic layer conditions.
vs AnyV2V / VideoAnyDoor: the former propagates single-image edits, while the latter inserts a static subject image into video; StM combines two existing videos and must preserve dynamics from both layers.
vs LayerFlow: LayerFlow generates layered videos from text, whereas StM recomposes existing layers; comparisons should account for differing conditioning information and control objectives.
Transferable insight: reconstruction-based editing tasks should examine whether conditions already reveal the answer; targeted disruption of such shortcuts may matter more than simply scaling reconstruction data.

Rating¶

Novelty: 4/5. Automatic decomposition supervision and composition-specific shortcut prevention form a clear methodological combination.
Experimental Thoroughness: 4/5. Comparisons, ablations, human and VLM judgments, and mask perturbations are covered, but test scale and quantitative multi-object evaluation remain limited.
Writing Quality: 4/5. The motivation and failure modes are clear, although some advantage claims require the context of ablations and subjective-preference exceptions.
Value: 4/5. The work provides reusable data construction and training strategies for dynamic layered composition, with scale, occlusion, and hard constraints still unresolved.