OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-based Video Editing¶

Conference: ECCV 2026
Paper: ECCV Official
Code: TBD
Area: Video Generation
Keywords: Instruction-guided Video Editing, Dataset Construction, Multimodal Large Language Model, Mixture of Experts, Benchmark Evaluation

TL;DR¶

Presents OpenVE-3M, a large-scale, high-quality dataset comprising 3 million instruction-video editing pairs across 8 categories, along with the human-aligned OpenVE-Bench and the 5B OpenVE-Edit model, which significantly outperforms existing 14B open-source models.

Background & Motivation¶

Propelled by advances in multimodal generative models, instruction-guided image editing has witnessed rapid progress with models like FLUX-Kontext, Qwen-Image-Edit, and Nano Banana. The success of these open-weight image models is heavily anchored on the availability of large-scale, high-quality paired training datasets. In stark contrast, instruction-guided video editing (IVE) remains severely constrained. Extending instruction editing from static 2D images to continuous spatiotemporal videos introduces immense challenges: edits must preserve spatial fine-grained layout and appearance changes while rigorously maintaining temporal consistency, motion coherence, and long-term foreground-background identity across frames.

Existing open-source IVE datasets suffer from four primary limitations: limited scale, scarce edit categories, brief instructions, and a high proportion of degraded samples. Datasets like InsViE-1M and Señorita-2M contain only 1M to 2M samples with average instruction lengths of merely 3 to 4 words—insufficient for guiding nuanced visual alterations. Meanwhile, VIVID-10M provides only binary video masks rather than actual edited videos, preventing end-to-end training. Furthermore, existing pipelines lack robust spatiotemporal filtering mechanisms, resulting in abundant noise, temporal flickering, and distribution drift. On the evaluation side, the community has also lacked a standardized, multi-category benchmark tightly aligned with human judgment.

This work bridges these critical gaps by building a comprehensive end-to-end framework encompassing data generation, model architecture, and benchmark evaluation. Core idea: Construct OpenVE-3M, a 3M high-quality IVE dataset spanning spatially-aligned and non-spatially-aligned tasks across 8 fine-grained categories with strict spatiotemporal and VLM-based filtering; develop OpenVE-Edit, a 5B architecture leveraging MLLM semantic extraction and a zero-initialized MoE-Connector; and establish the human-aligned OpenVE-Bench.

Method¶

Overall Architecture¶

The OpenVE framework consists of three closely coordinated stages: a three-stage industrial-grade data curation pipeline that consumed over 10,000 GPU-days to synthesize and filter 3M high-quality video pairs; an efficient 5B video editing model OpenVE-Edit that synergizes an MLLM, a task-aware MoE-Connector, and a Diffusion Transformer (DiT); and a comprehensive evaluation benchmark OpenVE-Bench.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Video Library & Preprocessing<br/>Open-Sora-Plan / OpenViD / UltraVideo"] --> B["Stage 1: Taxonomy-Guided Generation<br/>6 Spatially-Aligned + 2 Non-Spatially-Aligned types"]
    B --> C["Stage 2: Rigorous Multi-Metric Filtering<br/>TF/CLIP-F Temporal Filtering + InternVL3.5-38B Scoring"]
    C --> D["OpenVE-3M Dataset (3M Pairs)"]
    D --> E["Stage 3: Joint Multimodal Encoding<br/>Qwen2.5-VL extracts joint video-instruction semantics"]
    E --> F["Stage 4: MoE-Connector Task Decoupling<br/>Learnable Queries route to sparse specialized experts"]
    F --> G["Stage 5: Zero-Init Accelerated Convergence<br/>Zero-initialized linear layer concatenated with umT5 into DiT"]
    G --> H["OpenVE-Edit (5B) Edited Video Output"]

Key Designs¶

1. Taxonomy-Guided Generation: Tailored Multi-Model Pipelines for Heterogeneous Edit Tasks

Treating diverse editing demands with a homogeneous pipeline inevitably causes failure in either local object manipulation, camera transitions, or stylistic harmony. The paper formalizes eight categories under two branches: Spatially-Aligned (SA)—comprising Global Style, Background Change, Local Change, Local Remove, Local Add, and Subtitles Edit; and Non-Spatially-Aligned (NSA)—comprising Camera Multi-Shot Edit and Creative Edit. In SA tasks, FLUX-Kontext performs first-frame manipulation, while Canny edge sequences extracted from the original video act as structural conditioning for Wan2.1-Fun-14B-Control to synthesize motion-coherent videos. For local object addition, an ingenious reverse construction strategy is deployed: video inpainting models erase an object to create the source video, while the unedited video serves as ground-truth target for "Local Add". Conversely, synthetic object addition combined with segmentation pasting creates source videos for "Local Remove". For NSA tasks, Seedance's native multi-shot capabilities and creative text prompting produce smooth camera transitions and imaginative dynamic variations of consistent subjects.

2. Rigorous Multi-Metric Filtering: Temporal Heuristics and VLM Upper-Bound Verification

Synthetic generation pipelines inherently risk temporal flickering, visual artifacts, and distribution drift. A two-tier screening filter cleans all generated candidates. The first tier employs automated physical metrics: Temporal Flickering (TF, mean absolute pixel difference across frames) and CLIP-F (average inter-frame CLIP cosine similarity) to immediately reject unstable, flickering videos. The second tier conducts fine-grained multimodal evaluation using Vision-Language Models across three dimensions on a 1-5 scale: Instruction Compliance, Consistency & Detail Fidelity, and Visual Quality & Stability. Crucially, the protocol enforces an "Instruction Compliance Upper Bound"—no sample can receive a high overall score if it fails to follow the editing prompt, regardless of pristine visual appeal. Benchmarked against 300 human-annotated samples, InternVL3.5-38B was selected to score the complete corpus, discarding all pairs scoring below 3 to guarantee spotless dataset fidelity.

3. Joint Multimodal Encoding: Extracting Spatiotemporal Semantics via MLLM

Standard video diffusion architectures rely on standalone text encoders (such as umT5 or CLIP) whose text embeddings offer only surface-level lexical semantics without grounding to spatial video regions or temporal dynamics. OpenVE-Edit feeds both the source video frames and the natural language instruction directly into a lightweight MLLM (Qwen2.5-VL-3B). Through bidirectional self-attention in the MLLM backbone, the model comprehends high-level semantic, spatial, and temporal correspondences—explicitly discerning what to modify, what to keep invariant, and how actions unfold. Furthermore, an instruction token filtering mechanism strips extraneous prefix and system tokens, transmitting solely task-salient hidden states to downstream modules.

4. MoE-Connector Task Decoupling: Sparse Expert Routing Driven by Learnable Queries

Deploying shared linear projections across heterogeneous video editing tasks causes parameter competition and negative transfer. To resolve this, the MoE-Connector bridges the MLLM and the DiT. Addressing variable sequence lengths from differing video resolutions and frame counts, the connector uses 512 learnable decoder queries \(Q_l \in \mathbb{R}^{b \times L_q \times D_h}\) to extract task-relevant features via cross-attention. Inside the MoE-FFN, 6 distinct expert sub-networks are maintained, with a gating network dynamically selecting the top-2 experts per sample:

\[S = \text{topk}(\text{softmax}(W_g x), k), \quad \mathbf{y} = \sum_{i \in S} w_i \cdot \left( W_{i,2} \cdot \text{GELU}(W_{i,1} x + b_{i,1}) + b_{i,2} \right)\]

This sparse allocation isolates task-specific gradients (e.g., style alteration vs. geometric removal), ensuring parameter efficiency and specialized feature representations.

5. Zero-Init Accelerated Convergence: Preserving Pretrained Generative Priors

A randomly initialized connector module typically introduces noisy activations into the pretrained DiT at early training steps, disrupting pretrained diffusion priors and destabilizing convergence. Drawing inspiration from ControlNet, the final linear transformation layer \(W_o\) of the MoE-Connector is initialized to zero. At initialization, the connector output evaluates strictly to zero, leaving the concatenated umT5 instruction features entirely unaffected as they interact via cross-attention in the DiT. During backpropagation, \(W_o\) smoothly shifts away from zero, gradually and stably injecting the rich spatiotemporal visual semantics extracted by the MLLM without initial mode collapse or slow warm-up.

Loss & Training¶

The base generative backbone is Wan2.2-TI2V-5B. Optimization follows a two-stage regimen: Stage 1 trains for one epoch at 480P resolution with a global batch size of 512 and learning rate \(1 \times 10^{-5}\); Stage 2 fine-tunes for an additional epoch at 720P with learning rate \(1 \times 10^{-6}\). To overcome severe computational bottlenecks during data generation—where Wan2.1-Fun-14B-Control originally required 50 minutes per 81-frame 720P clip—the authors adopted SageAttention-2 and reduced inference steps from 50 to 10, achieving an \(8\times\) speedup (down to 6 minutes per video) without fidelity degradation, and employed 4-step distilled Wan2.2-I2V-A14B-NFE4-V1 for I2V tasks.

Key Experimental Results¶

Main Results¶

On OpenVE-Bench (431 video pairs across 8 categories), models are evaluated using Seed1.6-VL (1-5 scale) across all subcategories, alongside overall scores from Gemini 2.5 Pro and temporal metrics TF and CLIP-F:

Methods	#Params	#Reso	Overall (Seed1.6)	GS	BC	LC	LR	LA	SE	CE	CME	Overall (Gemini)	TF	CLIP-F
Runway Aleph (Closed)	-	1280×720	3.50	3.47	2.84	3.88	3.88	2.79	3.50	3.23	4.48	3.65	0.9851	0.9659
VACE	14B	1280×720	1.17	1.41	1.16	1.43	1.00	1.05	1.02	1.13	1.16	1.57	0.9852	0.9685
OmniVideo	1.3B	640×352	1.02	1.02	1.00	1.00	1.00	1.00	1.16	1.00	1.00	1.31	0.9916	0.9785
InsViE	2B	720×480	1.40	2.25	1.23	1.60	1.00	1.23	1.22	1.68	1.02	1.53	0.9768	0.9739
AnyV2V*	3.7B	512×512	1.41	1.84	1.31	1.94	1.09	1.27	1.02	1.57	1.11	1.51	0.9614	0.9589
Señorita*	5B	800×448	1.90	2.29	1.47	2.69	1.93	1.64	2.49	1.91	1.02	1.93	0.9802	0.9658
Lucy-Edit	5B	1280×704	1.95	2.17	2.20	3.30	1.03	2.37	1.06	2.35	1.14	2.15	0.9836	0.9692
ICVE	13B	384×240	2.25	2.35	1.86	2.91	2.68	2.27	2.04	1.94	1.38	2.07	0.9877	0.9661
DITTO	14B	832×480	2.06	3.70	2.23	2.28	1.00	2.08	1.01	2.61	1.51	1.98	0.9848	0.9635
VideoCoF	14B	1280×704	2.24	1.07	1.33	3.52	3.26	2.67	2.57	1.64	1.05	2.24	0.9768	0.9651
OpenVE-Edit (Ours)	5B	1280×704	2.41	3.11	2.72	3.19	1.42	2.41	2.56	2.01	1.24	2.49	0.9889	0.9698

Ablation Study¶

Table 3 investigates the incremental impact of each structural component and the choice of training dataset on OpenVE-Bench overall performance:

Index	MLLM Feature Injection	MoE-Connector	Training Dataset: Señorita-2M	Training Dataset: OpenVE-3M	Overall Score	Note
1	✗	✗	✗	✓	2.12	Baseline DiT with umT5 text features only
2	✓	✗	✗	✓	2.31	Added MLLM features via standard shared MLP (+0.19)
3	✓	✓	✓	✗	1.54	Full architecture trained on Señorita-2M (-0.87 drop)
4	✓	✓	✗	✓	2.41	Full OpenVE-Edit trained on OpenVE-3M (Best)

Table 4 assesses the correlation between automated VLM evaluations and human expert consensus (ICC = 0.948 across 300 samples):

Judge Model	PLCC	SROCC	Note
Seed1.6-VL	0.833	0.842	Highest human correlation; designated primary benchmark judge
Gemini 2.5 Pro	0.731	0.756	Strong alignment among commercial APIs; secondary judge
InternVL3.5-38B	0.670	0.643	Best open-source trade-off; selected for dataset filtering
Qwen3-VL-30B-A3B	0.583	0.555	Moderate alignment; utilized only for coarse screening

Key Findings¶

Data Quality Dictates the Upper Bound of IVE: Replacing OpenVE-3M with the prior benchmark dataset Señorita-2M on the exact same architecture causes performance to plummet from 2.41 to 1.54 (-36.1% relative drop), substantiating the severe adverse effect of low-quality, instruction-violating samples in prior datasets.
MLLM Semantic Grounding plus MoE Routing Yield Steady Gains: Feeding joint video and instruction inputs to the MLLM provides a +0.19 boost over text-only conditioning, while decoupling heterogeneous edit tasks with MoE further elevates the score by +0.10, showing superior spatial preservation and instruction compliance.
Structural Control Preferences in Data Generation: Canny edge control consistently outperforms depth estimation in preserving high-frequency textural details during dynamic camera motion; combined with SageAttention-2 and step reduction, it delivered an \(8\times\) throughput speedup essential for dataset scaling.

Highlights & Insights¶

Inverse Construction for Object Edits: By reversing object inpainting for addition and leveraging composition-pasting for removal, the data pipeline synthesizes photorealistic training pairs without suffering from temporal occlusion artifacts.
Instruction Compliance Upper Bound Metric: Enforcing instruction following as an absolute ceiling for quality ratings prevents models from gaining deceptive high scores by trivially repeating source frames.
Zero-Initialized Multimodal MoE Adapter: Demonstrates that Zero-Convolution concepts extend naturally to sequence-level MoE adapters, enabling smooth injection of dense visual-language representations into frozen video diffusion transformers.

Limitations & Future Work¶

Author-Admitted Limitations: OpenVE-3M lacks reference-based video editing pairs (e.g., editing subjects according to external reference photographs); constrained computing budgets restricted experimental verification to a 5B parameter envelope.
Identified Weaknesses: Extreme spatiotemporal modifications such as Local Removal (1.42) and Camera Multi-Shot Transitions (1.24) remain persistently challenging, highlighting that non-spatially-aligned video editing is still an open problem.
Future Directions: Exploring direct token-dimension noise concatenation in video latent spaces, and advancing towards unified models that seamlessly bridge video perception, autoregressive planning, and diffusion generation.

vs InsViE-1M / Señorita-2M: OpenVE-3M expands dataset scale to 3 million pairs, broadens task taxonomy to 8 categories, increases instruction length tenfold (40.6 vs 3.6-4.4 words), and applies rigorous multi-stage filtering.
vs DITTO / ICVE: DITTO is predominantly specialized for style transfer, while ICVE requires 13B parameters and suffers from context-length OOM on high resolutions; OpenVE-Edit achieves superior overall performance across all 8 categories with only 5B parameters.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Introduces the first 3M comprehensive IVE dataset bridging spatially and non-spatially aligned edits with an MoE-decoupled architecture.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-model comparisons against 10 baselines, exhaustive ablations, and rigorous human-VLM alignment validation.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, transparently detailing the complex multi-stage data curation and architectural innovations.
Value: ⭐⭐⭐⭐⭐ Provides invaluable, standardized open-source data and benchmarking assets to the video editing research community.