Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

Conference: ECCV2026
Paper: ECCV paper
Project: Goku
Area: Video Generation
Keywords: instruction-based video editing, structural editing, progressive filtering, dual-branch diffusion, spatial alignment

TL;DR

Goku constructs 2 million video editing pairs through task-specific synthesis and progressive filtering, while its mask-guided Goku-Edit model raises instruction-only IF on Goku-Bench from LucyEdit's 0.549 to 0.627 across appearance, structural, reference-based, and compositional editing.

Background & Motivation

Natural-language video editing involves more than changing clothing colors: users may reposition a subject, change a camera trajectory, or request several modifications at once. Existing paired datasets largely emphasize individual appearance changes. A model can therefore produce attractive frames without reliably identifying the region to modify, preserving everything else, or satisfying all sub-instructions.

The first obstacle is supervision. Direct object insertion can produce floating objects and incorrect occlusions, while framewise stylization can flicker. Changing subject actions or positions also requires more than replacing textures. Goku combines image editing, video generation, segmentation, and multimodal understanding to construct these pairs, with quality checks on source videos, generation conditions, and final outputs rather than only completed videos.

Beyond broader data coverage, the editor needs to connect where an edit occurs with what it should look like. Core Idea: establish complex editing supervision through task-specific synthesis and progressive quality gates, then assign spatial structure to a low-resolution mask branch that guides the high-resolution video branch through aligned bidirectional attention.

Method

Overall Architecture

The work has three layers: the Goku dataset, the Goku-Edit model, and Goku-Bench. The data pipeline selects 1 million source clips from Koala-36M, prepares instructions, temporal masks, and reference images, and ultimately produces 2 million editing pairs. Table 1 reports 720p videos with 65 to 129 frames per clip.

The 10 task classes comprise five basic appearance edits plus subject movement, camera movement, multi-task editing, reference addition, and reference replacement. Multi-task examples combine 2 to 5 tasks. At inference time, the model receives a source video, a text instruction, and an optional reference image; it produces the edited video while predicting an editing-region mask. Goku-Bench evaluates execution using 1,000 human-verified cases.

The diagram summarizes the data-to-model relationship. Progressive filtering is embedded before, during, and after synthesis rather than applied only once to a finished dataset.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source videos, instructions<br/>masks and reference images"] --> B["Task-Specific Synthesis"]
    B --> C["Progressive Filtering"]
    C --> D["Goku editing pairs"]
    D --> E["Dual-Branch Spatial Alignment"]
    E --> F["SpatialCFG"]
    F --> G["Edited videos and masks<br/>Goku-Bench evaluation"]

Key Designs

1. Task-Specific Synthesis: match each edit to suitable generation conditions

Preprocessing applies shot-transition detection, aesthetic scoring, motion analysis, and OCR-based watermark handling, followed by Gemini2.5-Pro content-richness screening. Clips are trimmed to 3 to 10 seconds. Gemini2.5-Pro generates structured instructions and object labels from video content, covering over 200 object categories, 100 styles, and 20 camera-motion patterns. Multi-task instructions must avoid conflicts, such as removing an object while adding something to it. Grounded-SAM2 extracts temporal masks from the labels, and reference images are prepared to match the target region's lighting and pose.

Basic editing does not use a single generator indiscriminately. Minimax-Remover produces removal examples; swapping the original and object-removed videos yields addition supervision, avoiding the occlusion problems of direct insertion. Replacement and attribute alteration feed the source video, masks, and Flux-generated reference images into VACE. For stylization, Flux first edits the initial frame, then VACE propagates that style using the stylized frame and a per-frame depth sequence to constrain scene geometry.

Structural edits require further specialization. For action changes, Gemini2.5-Pro generates two action descriptions for the same subject, and Wan2.2 synthesizes corresponding videos intended to preserve identity and background while varying action. For position changes, Flux relocates the object in the first frame, and Wan2.2 extends that frame into a video. RecamMaster handles camera movement, with complex trajectories decomposed into basic motion sequences. Multi-task edits execute sequentially, passing each output to the next step, so early errors can contaminate later operations.

Reference editing also needs protection against pixel copying. After cropping a target reference image from the video, the pipeline uses Flux to change pose, lighting, or background. This preserves the reference object's semantics while breaking exact pixel correspondence, encouraging the model to treat the image as an identity and appearance condition rather than a patch to paste back.

2. Progressive Filtering: stop bad conditions before expensive synthesis

The first tier removes low-quality source videos, yielding 1 million source clips shared across tasks. The second checks generation conditions: IoU thresholding assesses mask completeness, while Gemini2.5-Pro checks instruction-target semantic agreement and the plausibility of synthesized reference images. The main text does not specify the IoU threshold or its full computation protocol, so this cannot be presented as a fully reproducible numerical rule.

The third tier validates generated pairs. Low-level checks cover inter-frame consistency, frequency-domain artifacts, and aesthetic rescoring; Gemini2.5-Pro evaluates editing accuracy and photorealism. Approximately 88% of synthesized samples are rejected. The paper states that supplementary material contains human verification with 100 samples per task and 3 annotators, precision/recall analysis, and an open-source Qwen3VL-30B filtering alternative. Those detailed results are absent from this cache.

3. Dual-Branch Spatial Alignment: separate structure and appearance while exchanging state

Both branches adapt pretrained Wan2.2-5B. The primary video branch receives text, the source video, and a reference image; the auxiliary branch receives the same text and a spatially downsampled source video to predict the target editing-region mask. A frozen Qwen3VL-8B encodes instructions for both branches, and a pretrained VAE encodes images, videos, and masks. The main branch concatenates reference-image latents, source-video latents, and noisy edited-video latents along the temporal dimension. The mask branch temporally concatenates downsampled source-video latents and noisy mask latents; this is not channelwise concatenation.

The mask branch downsamples both height and width by \(n\), with default \(n=4\), so its grid indices cannot be compared directly with the video's. Before RoPE computation, mask positions \((x,y)\) are mapped to \((nx,ny)\) without changing the rotary frequency basis. Physically corresponding locations therefore avoid artificial large relative offsets caused by different grid scales; the remaining high-resolution offsets reflect actual within-cell distances.

Aligned features interact through bidirectional cross-attention. M2V uses video features as Query and mask features as Key/Value, injecting structural information into the video branch through a residual connection. V2M feeds the evolving video state back into the mask branch. This reverse direction matters: a mask predicted only from the downsampled source cannot track how the edited content changes during denoising, leading to mask-content disagreement and unstable boundaries.

4. SpatialCFG: strengthen cross-branch spatial signals during inference

Ordinary text CFG amplifies text conditioning rather than mask constraints. SpatialCFG contrasts coupled predictions, with bidirectional cross-attention enabled, against independent predictions obtained by disabling cross-branch attention and replacing the counterpart input with a null latent. Independent predictions retain the current branch's computation and text conditioning, making the difference more specific to the cross-branch contribution.

The following compact notation follows the prose explanation of Equation (3). The cached equation has extraction damage, so this is not a verbatim transcription:

\[ \tilde{\epsilon}_b=\hat{\epsilon}_b^{\mathrm{indep}}+s_b\left(\hat{\epsilon}_b^{\mathrm{dual}}-\hat{\epsilon}_b^{\mathrm{indep}}\right),\qquad b\in\{e,m\}. \]

\(e\) and \(m\) denote the video and mask branches. The video coefficient is intended to suppress unintended changes outside the edited region, while the mask coefficient strengthens boundary consistency. Spatial guidance is applied under both text-conditioned and text-unconditioned settings before ordinary text CFG, as described in Equation (4). It requires no additional training but entails extra prediction computation. The main text does not report complete latency measurements or specific guidance coefficients, so training-free should not be read as cost-free inference.

Loss & Training

The main text describes corresponding noise predictions from both branches but does not fully specify loss weights, optimizer, learning rate, total training steps, or hardware. These parameters are not supplied here, and two Wan2.2-5B-based branches should not simply be labeled a single 5B model.

The directly verifiable controlled setup is Table 3: the same LucyEdit architecture, 50k samples, 6k training steps, and Goku-Bench evaluation are used to isolate training-data effects, with an additional 100k Goku result. Figure 5 compares \(n=1,2,4,8\): \(n=1\) gives smooth but texture-poor clothing, \(n=4\) combines texture and localization, and \(n=8\) weakens structural signals excessively. The default is therefore \(n=4\).

Key Experimental Results

Main Results

Goku-Bench contains 1,000 cases from Koala-36M, with videos at least 720p and 3 to 10 seconds long, paired with manually written instructions. Beyond general video-quality metrics, the paper introduces 7 editing metrics. PR measures plausible physical interactions, SR measures instruction-specified spatial relations, IF measures instruction completion, and EQ measures overall naturalness and editing consistency. Gemini2.5-Pro evaluates these four using scoring rubrics.

Three metrics are task-specific: SuM uses Gemini2.5-Pro to assess the realism and fluency of subject-motion trajectories, a narrower focus than PR's overall physical interactions; CM uses optical flow to identify camera-motion types; ST measures DINO feature similarity between a reference style image and generated frames. Detailed judging prompts and complete aggregation rules are not included in the main-text cache.

The following selection from Table 2 covers instruction-only tasks. Lower FVD is better; higher is better for the other listed metrics.

Model	FVD	PR	SR	IF	EQ
Omni-Video	1032.08	0.580	0.631	0.510	0.590
LucyEdit	1420.36	0.476	0.755	0.549	0.579
Goku-Edit	993.93	0.738	0.832	0.627	0.645
Runway	1038.52	0.705	0.793	0.758	0.782
Luma	1095.64	0.681	0.769	0.741	0.761

Relative to LucyEdit, Goku-Edit improves IF by 0.078, or 7.8 percentage points, which is the careful interpretation of the abstract's approximately 8% claim. Its IF and EQ remain below both commercial models. On reference-image-plus-instruction tasks, Goku-Edit achieves IF 0.851 and FVD 925.55, whereas Runway achieves IF 0.872. These results should not be merged with instruction-only tasks into one ranking.

Ablation Study

Table 3 examines how training data affects multi-task editing. The first five rows use 50k samples and 6k steps with the same LucyEdit architecture; this is not a component ablation of the complete Goku-Edit model.

Data Configuration	Samples	FVD	IF	EQ
InsV2V	50k	3102.15	0.298	0.241
InsViE	50k	2843.67	0.301	0.253
Señorita	50k	2761.44	0.289	0.278
Goku, unfiltered	50k	2512.08	0.318	0.331
Goku, filtered	50k	1380.45	0.501	0.522
Goku, filtered	100k	1241.33	0.519	0.538

At fixed 50k scale, filtering raises IF from 0.318 to 0.501, a gain of 0.183; scaling to 100k raises it further to 0.519. The former supports quality control, while the latter only demonstrates additional scale benefits over the tested range, not an unlimited scaling law.

Table 4 adds components sequentially. Adjacent gains should not be interpreted as independent, freely additive component effects.

Cumulative Configuration	PR	SR	IF	EQ
Baseline	0.651	0.743	0.541	0.578
+ MLLM	0.664	0.756	0.573	0.591
+ Dual branch	0.695	0.789	0.588	0.612
+ RoPE alignment	0.718	0.819	0.608	0.631
+ SpatialCFG	0.738	0.832	0.627	0.645

Key Findings

• Replacing the text encoder raises IF by 0.032, but the later spatial mechanisms remain necessary: stronger language understanding alone does not resolve localization and boundary errors.
• RoPE alignment raises SR from 0.789 to 0.819; SpatialCFG subsequently adds 0.019 IF, supporting the roles of cross-scale correspondence and inference-time spatial enhancement.
• In Table 5, 30 participants each rate 100 videos on a 5-point scale. Goku-Edit scores 4.58/4.51/4.65 for IF/video quality/temporal consistency, against LucyEdit's 4.05/4.12/4.08. Commercial models are absent from this human-study table, so it cannot establish superior human preference over commercial systems.

Highlights & Insights

• Reversing removal pairs into addition supervision is a transferable data-construction strategy. It solves the more stable deletion problem first instead of relying on difficult direct insertion.
• The mask is a jointly evolving generated variable rather than a static input. Bidirectional feedback adapts structural constraints to the current edit instead of only injecting source-derived masks into the video branch.
• Cross-resolution attention requires more than matching feature dimensions: positional encodings must share a physical coordinate system. RoPE index scaling is one of the work's clearest reusable mechanisms.

Limitations & Future Work

• The main text has no dedicated limitations section. The following points are reading-based assessments, not statements presented as the authors' explicit admissions.
• Synthesis and filtering depend on teachers including Gemini2.5-Pro, Flux, VACE, and Wan2.2; rejecting approximately 88% of candidates also implies substantial generation cost. Detailed equivalence of the open-source filtering alternative is absent from this cache.
• Gemini2.5-Pro participates in instruction generation, filtering, and several automatic metrics, creating a risk of shared judging preferences. Training and test sources both come from Koala-36M, and the main text does not fully specify deduplication. This is not evidence of confirmed leakage.
• Subject-action pairs rely on a generator to preserve identity and background. Filtering reduces inconsistency risks but does not prove that every unedited factor remains fixed in every pair.
• Commercial models still lead on IF and EQ, while complete training hyperparameters, latency, memory use, and guidance coefficients are missing from the main text. Independent judges, cross-source testing, and cost-quality curves would strengthen the evidence.
• Table 1 marks OpenVE-3M as supporting camera movement, in tension with prose describing it as appearance-only. The safer distinction is Goku's combined coverage of subject movement, reference editing, and 2 to 5 composed tasks, not exclusive support for camera movement.

Related Work & Insights

• Compared with Ditto / Señorita-2M / InsViE: the emphasis is task coverage and filtering, not sample count alone. Table 3 supports this distinction with a shared training architecture, but only within its specified multi-task setting.
• Compared with VACE / AnyV2V: first-frame editing followed by propagation and mask-conditioned generation serve as synthesis tools. Goku additionally converts such supervision into a trainable general-purpose editor.
• Compared with LucyEdit: LucyEdit is both a main-table baseline and the common architecture in data ablations. Evidence that Goku data improves LucyEdit should be distinguished from evidence that Goku-Edit's architecture improves the final system.

Rating

• Novelty: 4/5. Complex-task coverage and cross-scale dual-branch alignment form a distinct combined contribution.
• Experimental Thoroughness: 4/5. Data, component, and human evaluations are provided, but reproducibility parameters and cost reporting are incomplete.
• Writing Quality: 3/5. The methodological narrative is clear, but scope claims are inconsistent with the comparison in Table 1.
• Value: 4/5. The work provides useful task definitions, data-construction ideas, and evaluation dimensions for structural and compositional video editing.