MSEditor: Toward Consistent Multi-Shot Video Editing¶

Conference: ECCV2026
Paper: ECCV 2026
Area: Video Generation
Keywords: multi-shot video editing, cross-shot consistency, diffusion model, subject preservation, sparse cross-attention

TL;DR¶

MSEditor is the first framework built specifically for multi-shot video editing: it repurposes multi-view video datasets into multi-shot supervision with geometrically consistent masks, injects that cross-shot prior into a Wan2.1 diffusion backbone through a zero-initialized Supervisory Adapter, and lets semantically related shots exchange features inside one attention window via Cross-Shot Packing and Sparse Cross Attention, so that an edit made on the first frame propagates consistently across all shots in a single inference pass.

Background & Motivation¶

Multi-shot video editing serves real content production — films, narrative shorts, commercials. Such content is inherently assembled from discontinuous shots, and both the cutting rhythm and the narrative tension come from switching between them. Yet almost every existing video editing method is single-shot: in the UNet era TokenFlow propagated features across frames and Video-P2P controlled cross-attention; in the DiT era VACE, Ditto and VideoPainter rely on unified conditioning and synthetic training data. All of them share one premise — that a dense frame-to-frame correspondence exists along a single continuous timeline, so consistency can be maintained by temporal attention inside a short video window. That premise simply does not hold for multi-shot sequences.

Moving these methods onto multi-shot footage runs into two walls at once. The first is data: large-scale, high-quality training data containing the same subject across several discontinuous shots barely exists, and collecting it is prohibitively expensive, so current frameworks never see shot-level semantic correspondence during training and fall back on short-term temporal correlation. The second is error propagation and inference cost: a multi-shot sequence jumps abruptly in viewpoint, camera scale (from a wide establishing shot to a close-up) and subject pose. When each shot is edited independently, small artifacts and identity drift in an early shot amplify recursively in later ones, showing up as visibly inconsistent subject appearance, style and structure across shots. Worse, once the sequence exceeds the model's temporal context (about 81 frames for Wan2.1) the sequence must be split into chunks that are inferred separately — multiple forward passes are slow, and each chunk loses any cross-shot semantic awareness.

Behind both walls lies the same core tension: cross-shot consistency requires the model to see and share subject semantics between shots, yet the prevailing paradigm treats every shot as an unrelated sample — concatenated along the batch dimension at training time and handled as an independent video at inference time — so cross-shot information has no channel to flow through at any stage. This paper's angle is to open that channel twice. During training it uses multi-view video datasets as a proxy for multi-shot supervision, treating synchronized multi-view sequences of one subject as natural "multi-shots" and learning that prior into the backbone through a zero-initialized adapter. During generation it dynamically packs semantically related shots into the self-attention window and computes them jointly, so different shots exchange features within a single attention operation. Core idea: recast multi-view datasets into geometrically consistent multi-shot supervision, and establish cross-shot information channels at both training and inference time via a Supervisory Adapter and Cross-Shot Packing, so that one inference pass propagates an edit consistently across all shots.

Method¶

Overall Architecture¶

The input is a multi-shot video together with editing masks; the output is an edited multi-shot video that keeps subject identity, motion and structure consistent across shots. The method is built on the pretrained diffusion transformer video backbone Wan2.1 (14B) and proceeds in two phases, "creating supervision" and "training the model." First, existing multi-view video datasets are repurposed: each sequence supplies synchronized RGB from 10 viewpoints plus depth and camera parameters, and a subject mask segmented completely in one shot is reprojected into another shot's view using depth and camera geometry to render a cross-shot consistent mask. Qwen3-VL then generates structured prompts consisting of one global caption plus one shot caption per shot. During training, the Supervisory Adapter injects the multi-view RGB and masks into the backbone through a zero-initialized ControlNet-style branch; in the self-attention layers Cross-Shot Packing concatenates frames of shots that are semantically related to an anchor shot into a single sequence; and in the cross-attention layers Sparse Cross Attention makes each shot's visual queries attend only to the global text and to that shot's own text. At inference there is no per-shot multi-pass generation: the user edits the first frame and supplies a mask, and a single forward pass propagates the change to every shot.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["multi-shot input video + editing masks"] --> B["Multi-view Reprojection Data Construction<br/>cross-shot consistent masks + structured prompts"]
    B --> C["Supervisory Adapter<br/>cross-shot prior injection + first-frame guidance"]
    C --> D["Cross-Shot Packing<br/>semantically related shots share self-attention"]
    D --> E["Sparse Cross Attention<br/>global + shot-level text conditioning"]
    E --> F["output: consistent multi-shot edited video"]

Key Designs¶

1. Multi-view Reprojection Data Construction: replace unreliable segmentation labels with geometric consistency

Multi-shot editing training data does not exist, so repurposing multi-view datasets as a proxy is a natural move — but editing is far more demanding than generation: it requires the subject to be cleanly disentangled from the scene, i.e. clean masks. Calling off-the-shelf segmentation models (SAM2, Grounded-SAM) fails in complex scenes. When the background contains distracting content such as paintings or incidental passers-by, the model may segment the subject successfully in one shot and fail in another. Supervision built from such "right in one shot, wrong in the next" labels teaches the model to drift across shots — worse than having no labels at all.

The paper's substitute hands the labeling problem to geometry rather than semantics. A mask that is complete in one shot is taken together with that shot's depth map and camera parameters, and double-reprojection (following prior work [80]) lifts it back into 3D and re-renders it from another shot's viewpoint. The projection depends only on geometry, so the reprojected mask corresponds to the source mask in a strict 3D sense: it is consistent across views by construction, bypassing the hit-or-miss behaviour of single-view segmentation. For semantics, an automated structured-prompting pipeline built on Qwen3-VL produces one global caption per sequence describing the overarching subject identity and scene, plus one shot caption per viewpoint describing that camera's angle and motion. The resulting training set has 3,400 video sequences, each with synchronized RGB from 10 viewpoints, corresponding segmentation masks, constructed depth maps and camera parameters. This hierarchical annotation — global semantics plus shot-level dynamics — is exactly what the Sparse Cross Attention design later relies on.

2. Supervisory Adapter: inject the cross-shot prior through a zero-initialized branch and anchor the edit intent on the first frame

A backbone such as Wan2.1 was trained on single-shot video only; it has no notion that "the same person or the same car appears in different shots," and full fine-tuning would damage its original generation and editing ability. Following ControlNet, the paper attaches a parallel set of adapter layers to the diffusion backbone: they take multi-view RGB videos and their segmentation masks as input and inject their output into the corresponding backbone layers, letting the backbone learn an implicit identity-consistent representation from "same subject, different viewpoints" supervision. Every adapter layer is initialized to zero, so it perturbs nothing at the start of training, which both prevents over-conditioning and keeps convergence stable.

On the conditioning side, the framework adopts the first-frame editing paradigm of recent video editing work so that users can state their edit intent explicitly. During training the mask of the first frame of the first shot is set to zero (that frame is not editable), which turns the first frame into guidance for generation — it carries the edit intent and acts as a high-fidelity visual anchor — while the remaining masks mark the regions to be rewritten in each shot. At inference the user edits the first frame once and provides the mask, and the change propagates through the whole multi-shot sequence. The training objective minimizes the mean squared error between predicted and ground-truth frames over all shot–mask pairs:

\[\mathcal{L}=\frac{1}{K}\sum_{k=1}^{K}\left\|\mathbf{V}^{(k)}_{\text{pred}}-\mathbf{V}^{(k)}_{\text{gt}}\right\|^{2}\]

(⚠️ Equation 3 is corrupted in the extracted text; this is reconstructed from the surrounding prose — refer to the original paper.)

This design carries only one half of the responsibility, namely injecting the prior at training time. In the ablation, removing first-frame editing drops the aesthetic score merely from 6.05 to 6.01 and cross-shot subject consistency from 0.9370 to 0.9321, and qualitatively the subject is still replaced correctly with only subtle deviations in fine details. Cross-shot consistency therefore stems mainly from implicit correspondences learned by the architecture, while the first frame contributes user control and detail fidelity rather than consistency itself.

3. Cross-Shot Packing: pack semantically related shots into one self-attention window

Even with a cross-shot prior in the backbone, editing degrades as the number of shots grows: the subject varies greatly in scale, position and appearance from shot to shot, and small deviations in early shots accumulate along the sequence. Standard training pipelines concatenate shots along the batch dimension — efficient, but it isolates shots completely inside the attention layers and severs the temporal and semantic relations between them; concatenating all shots along the frame dimension would give dense inter-shot interaction, but the memory and compute cost is prohibitive.

The compromise is dynamic grouping. Each training step randomly selects one shot as the anchor and keeps its full temporal resolution as the primary training target. For the remaining candidate shots, DINOv2 embeddings are used to measure semantic similarity to the anchor, and only the top-M shots above a threshold τ (e.g. 0.8) are kept; from each selected shot a representative set of n frames (e.g. the initial frames) is extracted as identity frames and concatenated after the anchor to form the packed sequence. Self-attention then performs feature interaction directly among semantically related shots, letting the model build a holistic understanding of subject identity across discontinuous segments, at the cost of only a few reference frames. To keep training feasible under different GPU memory budgets, a dynamic thresholding strategy is layered on top: if the packed sequence's total frame count exceeds the hardware capacity, τ is automatically raised (for instance from 0.8 to 0.85) to filter out the less relevant shots.

The design turns "which shots should look at each other" into a dynamic subset-selection problem governed by semantic similarity, and a handful of identity frames is enough to carry identity information from the anchor shot to the others. It is also the single largest contributor to cross-shot appearance consistency among the three ablated components: removing it drops cross-shot subject consistency from 0.9370 to 0.8919 and cross-shot background consistency from 0.9447 to 0.9132, with visible facial distortion in close-up shots.

4. Sparse Cross Attention: give global semantics and shot-level instructions their own places

Once the structured prompt is concatenated into one long text, ordinary dense cross-attention lets every video frame attend to every text token at once: the model struggles to disentangle which sentence describes which shot, so shot-level instructions never align with their own temporal segment. In the ablation, replacing this module with standard cross-attention makes the model fail outright on the second shot — a direct symptom of that semantic confusion.

The paper reuses the Sparse Cross Attention mechanism from HoloCine, splitting the text tokens explicitly into a global caption and per-shot captions so that the visual queries of shot \(i\) attend only to the concatenation of the global key–value pairs and that shot's own key–value pairs:

\[\mathrm{Attn}_{\text{cross}}(Q_i)=\mathrm{Attention}\big(Q_i,\,[K_{\text{global}};K_i],\,[V_{\text{global}};V_i]\big)\]

(⚠️ Equation 7 is partly corrupted in the extracted text; this is reconstructed from context — refer to the original paper.) Each shot's generation is thus constrained by global scene and subject semantics, which maintains cross-shot consistency, while retaining its own shot-level textual control, which maintains per-shot controllability. Consistency is delegated to the global keys and values and controllability to the shot-specific ones, so the two no longer interfere inside one dense attention map.

A Worked Example¶

Take the four-shot "sea turtle becomes a Ninja Turtle" sequence in Fig. 1. On the data side, the training sample comes from a multi-view dataset: one sequence provides RGB, masks, depth and camera parameters from 10 synchronized viewpoints, and the structured prompt consists of one global caption (subject identity plus scene) and four shot captions (camera angle and action per shot). For packing, shot 2 is randomly chosen as the anchor and keeps all of its frames. DINOv2 embeddings measure its similarity to shots 1, 3 and 4; if shots 1 and 3 score above τ = 0.8 while shot 4 falls below, the identity frames of shots 1 and 3 are appended to shot 2 and enter self-attention together to exchange features across shots. If the resulting frame count still exceeds this step's memory budget, the threshold is raised to 0.85 and the weaker of the two is filtered out again. On the text side, shot 2's visual queries see only the concatenation of the global caption and shot 2's caption; text from other shots never interferes. At inference the user repaints the turtle as a Ninja Turtle on the first frame and supplies a mask, and a single forward pass makes that edit hold simultaneously across all four shots without drifting in appearance, motion or structure.

Loss & Training¶

The training objective is the per-shot MSE reconstruction loss above, applied to all shot–mask pairs, with the first frame of the first shot masked out so that it serves as a non-editable guidance frame. All adapter layers are zero-initialized and trained together with the backbone. Optimization uses AdamW with a weight decay of 0.01 and an initial learning rate of \(1\times10^{-4}\); the input resolution is 480×832, the batch size is 8, and training runs on 8 NVIDIA A800 GPUs. Inference uses a flow-matching scheduler with 50 sampling steps. The dynamic thresholding described earlier turns the memory constraint into an automatic adjustment of the packing threshold τ.

Key Experimental Results¶

Main Results¶

Since no multi-shot video editing method existed before, all baselines come from single-shot editing and span both the UNet and DiT lines of work: TokenFlow, InsV2V, Ditto and VACE. Evaluation uses a purpose-built test set of 600 multi-shot internet videos (35% humans, 35% animals, 30% objects) covering style transfer, object replacement and attribute editing, with prompts generated automatically by Qwen3-VL; there is no overlap with the training data. Metrics are the Aesthetic Score (LAION aesthetic predictor), inter-shot and intra-shot consistency (subject/background), and semantic consistency (global/shot), the latter three following the evaluation protocol of HoloCine.

Method	Aesthetic ↑	Inter-shot·Subject ↑	Inter-shot·Background ↑	Intra-shot·Subject ↑	Intra-shot·Background ↑	Semantic·Global ↑	Semantic·Shot ↑
TokenFlow	5.51	0.9181	0.9279	0.9029	0.9173	0.1302	0.1415
InsV2V	4.89	0.9013	0.9307	0.8962	0.8941	0.1433	0.1137
VACE	5.91	0.9237	0.9372	0.8805	0.9365	0.1649	0.1738
Ditto	5.73	0.9287	0.9341	0.9125	0.9331	0.1514	0.1569
Ours	6.05	0.9370	0.9447	0.9501	0.9462	0.1912	0.1886

To rule out the explanation that the gain is purely a data effect, the paper also retrains the recent VACE and VideoPainter on its own multi-shot training data and compares under the same protocol:

Method	Aesthetic ↑	Inter-shot·Subject ↑	Inter-shot·Background ↑	Intra-shot·Subject ↑	Intra-shot·Background ↑	Semantic·Global ↑	Semantic·Shot ↑
VACE (retrained)	5.93	0.9301	0.9395	0.9057	0.9382	0.1728	0.1753
VideoPainter (retrained)	6.01	0.9237	0.9396	0.9061	0.9377	0.1687	0.1732
Ours	6.05	0.9370	0.9447	0.9501	0.9462	0.1912	0.1886

Ablation Study¶

Each of the three components is removed in turn and re-evaluated on two-shot and four-shot editing, with all other experimental settings held fixed:

Config	Aesthetic ↑	Inter-shot·Subject ↑	Inter-shot·Background ↑	Intra-shot·Subject ↑	Semantic·Shot ↑	Note
Full model	6.05	0.9370	0.9447	0.9501	0.1886	full model
w/o First Frame editing	6.01	0.9321	0.9357	0.9336	0.1729	generates from text and cross-shot priors only; smallest drop
w/o Cross-Shot Packing	5.83	0.8919	0.9132	0.8937	0.1521	degenerates to concatenating shots along the batch dim; largest loss in cross-shot consistency
w/o Sparse Cross Attention	5.70	0.8931	0.9256	0.8651	0.1432	standard dense cross-attention; semantic consistency collapses

Key Findings¶

Cross-Shot Packing is the most critical module for cross-shot appearance consistency. Removing it drops inter-shot subject consistency by 0.0451 (0.9370→0.8919) and inter-shot background consistency by 0.0315, with facial distortion appearing in close-up shots. "Shots must be able to see each other" is a necessary condition for avoiding identity drift; a training-time cross-shot prior alone is not enough.
Sparse Cross Attention affects semantic alignment rather than appearance. Without it, semantic-shot consistency falls by 0.0454 (0.1886→0.1432) and intra-shot subject consistency by 0.0850 (0.9501→0.8651), while the aesthetic score drops to 5.70, the lowest of the three ablations. Qualitatively the second shot fails to be edited at all, because dense attention lets every frame see every text token and the long structured prompt cannot be disentangled into "which sentence belongs to which shot."
First-frame editing is a visual anchor, not the source of consistency. Removing it costs only 0.04 in aesthetic score (6.05→6.01) and causes small declines across the consistency metrics; qualitatively the subject is still replaced correctly, with only subtle deviations in fine details. The authors argue from this that the core cross-shot capability comes from the architecture (adapter, packing, sparse cross-attention), while the first frame provides high-fidelity control and a one-touch user interaction — the only entry point through which a user acts once.
The advantage comes from architecture rather than data. VACE and VideoPainter retrained on exactly the same multi-shot data (Table 2) still lag behind MSEditor across the board, with the gap in semantic consistency especially clear (VACE 0.1728 and VideoPainter 0.1687 vs. 0.1912): what the single-shot paradigm lacks is not data but a cross-shot modeling mechanism.
The supplementary material additionally reports a quantitative evaluation of mask extraction accuracy, a "w/o mask" setting, a shape-editing task and user-study results, none of which appear in the main paper.

Highlights & Insights¶

Using multi-view datasets as a multi-shot proxy and labeling with geometric reprojection. The deadlock of "no multi-shot data" is resolved by switching both the data source and the labeling method; because the labels come from depth plus camera double-reprojection rather than semantic segmentation, the hit-or-miss behaviour of single-view segmentation is avoided at the root. The idea transfers to any task needing consistent cross-view supervision, such as multi-camera driving-video editing or cross-view 3D-consistent generation.
Cross-shot consistency is split into "inject a prior at training time" and "exchange information at inference time." The adapter writes the notion that different shots may show the same subject into the weights, while packing decides which shots should exchange features in this particular forward pass. This two-level structure is more controllable than simply extending the context window or simply scaling data, and it generalizes to long video generation and multi-subject editing.
Dynamic packing driven by DINOv2 similarity and a threshold. Deciding which shots to compute jointly becomes a semantically driven subset-selection problem, and the automatically raised threshold τ translates a memory constraint into a semantic trade-off knob — clean engineering that avoids the waste of a fixed window.
Structured prompting paired with sparse cross-attention. The global caption governs consistency and shot captions govern controllability, and their key–value pairs are separated explicitly at the attention level. Dispatching conditions of different granularity to different KV subsets is a pattern worth reusing wherever global coherence and local controllability must coexist.

Limitations & Future Work¶

The training data consists of multi-view datasets (10 synchronized viewpoints per sequence, mostly synthetic or controlled-capture scenes), which leaves a clear domain gap to real film footage: real multi-shot material is often shot asynchronously, with handheld camera work and abrupt cutting rhythms. No quantitative evaluation on real film clips is provided, so cross-domain generalization remains unverified.
The evaluation protocol is under-specified. The main paper only states that inter-shot and intra-shot consistency follow the protocol of prior work [51], without giving the feature extractor, the similarity definition or the aggregation; the user study, mask accuracy and the no-mask setting all sit in the supplementary material, which limits reproducibility and cross-paper comparison.
The method depends on user-provided editing masks, so mask quality directly affects the result. The claim that reprojected masks are more reliable is only verified in the training-data construction stage, with no evidence for hand-drawn masks at inference time.
Cost and availability: a 14B backbone, 8×A800 training and 50-step flow-matching inference, with no code or weight release mentioned, make real deployment expensive.
Concrete improvements: extend "anchor shot plus similar shots" into a learnable shot-relation graph so that packing adapts during training instead of relying on a fixed DINOv2 threshold; replace the global MSE with finer-grained per-shot consistency losses that directly constrain identity features between shots; and grow the training data beyond multi-view datasets by automatically organizing real film footage.

vs TokenFlow / InsV2V / Ditto: all three are single-shot editing routes (TokenFlow uses training-free cross-frame feature propagation; InsV2V and Ditto are instruction-based editors trained on synthetic data). Applied shot by shot to multi-shot sequences they have no channel between shots, and empirically they trail across aesthetic and consistency metrics, sometimes failing to perform the requested edit at all.
vs VACE / VideoPainter: these are stronger unified DiT-based video editing frameworks, but their training paradigm still treats each shot as an independent sample. Retrained on identical data they remain behind, which shows the gap lies in cross-shot modeling itself.
vs HoloCine: it also targets multi-shot narratives and cross-shot consistency and also uses sparse attention for long context, but it generates from scratch rather than editing and depends on auxiliary geometric inputs. This paper borrows its Sparse Cross Attention as the text-conditioning module to solve "how a structured prompt aligns with its own shot."
vs ShotAdapter / Long Context Tuning: these multi-shot generation methods maintain narrative coherence through transition tokens or an extended attention context, without editing control or mask propagation; this paper addresses unified rewriting of existing footage, a task setting orthogonal to theirs.

Rating¶

Novelty: ⭐⭐⭐⭐ It defines multi-shot video editing as a task and delivers the first dedicated framework, with genuinely new ideas on both the data side (multi-view repurposing) and the generation side (semantic packing); the main components (a ControlNet-style adapter and HoloCine's sparse cross-attention) are largely migrations of existing mechanisms.
Experimental Thoroughness: ⭐⭐⭐⭐ A main comparison, a same-data retrained comparison, a three-module ablation and rich qualitative results are all present, but the definitions of the core consistency metrics, the user study and mask accuracy live in the supplementary material, which weakens reproducibility.
Writing Quality: ⭐⭐⭐ The motivation and the two bottlenecks are clearly dissected and the three contributions are easy to follow; points are lost because the formulas are largely corrupted in the extracted text and key settings such as the mask convention and the scope of edit propagation are described only briefly.
Value: ⭐⭐⭐⭐ It offers a training-paradigm-level solution to cross-shot consistency and has direct value for multi-shot content generation and editing in film and advertising.