VOID: Video Object and Interaction Deletion¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: Object removal, counterfactual editing, quadmask, flow-warped noise, physical consistency

TL;DR¶

VOID extends video object removal from spatial inpainting to downstream interaction rewriting through counterfactual paired training, VLM-guided quadmasks, and an optional flow-warped noise second pass, receiving 64.8% of selections in a seven-way human evaluation on real videos.

Background & Motivation¶

Traditional video inpainting mainly restores occluded backgrounds, while effect-removal methods additionally erase shadows and reflections. However, these approaches often assume that the remaining scene motion should stay unchanged: later dominoes keep falling after intermediate blocks disappear, or a ball still stops where a removed person used to catch it. An edit can therefore look seamless while violating the scene's causal relationships.

Generative Omnimatte already uses generative priors to remove objects and their visual effects, but layered removal supervision does not sufficiently teach a model to change other objects' motion. Text-based editing also struggles because a user cannot easily specify every affected location over time, and a model may not infer which objects should stop, fall, or change direction. The desired output is not hidden content recovered from the original footage, but a new video showing what would happen without the object.

VOID consequently changes both supervision and the inference interface: re-simulation supplies targets with genuinely changed motion, while a VLM identifies where rewriting is allowed instead of leaving the diffusion model to guess across the entire frame. Core Idea: identify the regions requiring counterfactual changes, synthesize new motion, and then stabilize appearance along that motion, linking causal scope prediction to pixel generation.

Method¶

Overall Architecture¶

The input is a video and a binary mask of the object to remove; the output should eliminate that object, update its interactions plausibly, and preserve unaffected regions. During training, counterfactual paired supervision supplies before-and-after videos; at inference, VLM-guided quadmasks delimit edits, first-pass counterfactual generation proposes new motion, and optional flow-warped noise stabilization reduces deformation. Physical plausibility comes from simulated supervision and model priors, not an explicit physics solver operating on real videos at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Data["Counterfactual paired<br/>supervision"] -->|training| First["First-pass counterfactual<br/>generation"]
    Input["Video and object mask"] --> Mask["VLM-guided quadmasks"]
    Mask --> First
    First -->|substantial motion change| Refine["Flow-warped noise<br/>stabilization"]
    First -->|no second pass needed| Output["Counterfactual video"]
    Refine --> Output
    Data -->|training| Refine
    Mask --> Refine

Key Designs¶

1. Counterfactual paired supervision: re-evolve the scene after removal

For Kubric, the authors sample initial positions and velocities for multiple objects and simulate collisions, falling, and structural support, then remove selected objects while preserving all other initial conditions and re-simulating. The target therefore contains genuinely different subsequent trajectories, rather than simply hiding a layer from the original video. They generate approximately 1900 video pairs covering rigid-body interactions.

HUMOTO supplies motion-capture data of human-object interactions, with simulation and rendering performed with and without the human to generate approximately 4500 pairs. This adds cases such as manipulated objects losing support after the person disappears; human, object, and background-wall textures are randomized. Both datasets also randomize camera trajectories and focal zoom to help separate camera changes from object effects. These supervision sources are complementary because collision propagation and human manipulation have different motion distributions.

2. VLM-guided quadmasks: cover original positions, future positions, and overlap

Earlier trimasks often mark most of a frame as editable, providing weak spatial guidance; they also cannot clearly encode a pixel occupied both by a person to erase and by a ball that should subsequently pass through it. VOID separately represents the removal target, affected areas, and their intersection, distinguishing content that must disappear from locations where new content must still be generated. During training, rendering engines determine affected areas, which are then gridified to resemble the inference representation.

For real videos, SAM 2 first turns sparse user clicks into the target mask. The VLM examines the video and mask, describes objects affected by removal, and SAM 3 grounds those descriptions as masks of their original locations. That mask is then fed back to the VLM, with a coarse grid overlaid on the input video, to predict which cells could contain the affected objects in each counterfactual frame. This predicts coarse spatial regions rather than exact physical trajectory coordinates.

The original-position and counterfactual-position masks are united so that editing a new location does not leave the object behind at its old location:

\[ M_a = M_a^{\mathrm{orig}} \lor M_a^{\mathrm{count}}. \]

Here, \(M_o\) denotes the removal target and \(M_a\) the complete affected area. Pixels exclusively in \(M_o\) are black; pixels in both \(M_o\) and \(M_a\) are dark gray; pixels exclusively in \(M_a\) are light gray; all remaining pixels are white. Black requests removal, dark gray permits interaction content to be rebuilt after target removal, light gray permits affected-object changes, and white indicates regions to preserve. The mask is thus not merely segmentation: it is the spatial interface transmitting high-level counterfactual reasoning to the generator.

3. First-pass counterfactual generation: propose new motion within the editing scope

The generator uses a 5B CogVideoX diffusion Transformer initialized from Generative Omnimatte weights, inheriting a prior for separating objects and effects. It is fine-tuned on the new paired data with quadmask conditioning and receives the original video, mask, and Gaussian diffusion noise to predict a complete edited video. An unsupported object should start falling, for example, while an unobstructed object should continue moving instead of retaining its original motion by default.

This pass generates a motion hypothesis; the mask does not prescribe a unique trajectory. However, once the new motion departs from the input, the original frames no longer provide reliable geometric constraints, and a relatively lightweight video model can bend, stretch, or structurally alter rigid objects across frames. Plausible motion direction and stable object identity are therefore separate challenges, and the first pass does not guarantee the latter.

4. Flow-warped noise stabilization: preserve appearance along the new trajectory

Following Go-with-the-Flow, the second pass extracts optical flow from the first-pass output and constructs temporally correlated warped noise. Compared with temporally uncorrelated noise, organizing noise along the same motion trajectories encourages consistent denoising and reduces structural drift. Crucially, the flow comes from the prediction with changed motion, not from the input video, which would constrain the object toward its obsolete trajectory.

This pass uses a separately trained warped-noise variant of VOID, still conditioned on the original video and the same quadmask. It is not a generic training-free post-processing filter: training uses the same counterfactual pairs but derives noise from the target counterfactual video's optical flow. When generating the quadmask, the VLM also predicts whether removal causes substantial motion reconfiguration, such as free fall or trajectory changes, and triggers the second pass only in those cases. Its main role is to stabilize the proposed motion; an incorrect first-pass hypothesis should not be assumed to become causally correct through refinement.

A Worked Example¶

Consider removing the child catching a ball in Figure 3: SAM 2 obtains the child's mask, the VLM identifies the ball as affected, and SAM 3 locates the ball in the original footage. Using per-frame grid predictions, the VLM estimates where the ball would travel without being caught and combines those regions with the original ball locations. Where the child's upper body overlaps the ball's new path, the quadmask should be dark gray rather than removal-only black.

The first pass generates a counterfactual in which the ball continues moving and reaches the ground; if the ball deforms, flow along this new trajectory determines the second-pass noise. The intended result removes the child and changes the ball's trajectory while keeping unaffected regions stable. This example explains the paper's mechanism and is not an additional measured single-case success rate.

Loss & Training¶

The main text distinguishes the first-pass model using ordinary Gaussian noise from the second-pass variant using target-flow-aligned noise; both use the same data sources and quadmask conditioning. The available cache contains no appendix and does not provide verifiable learning rates, training steps, resolution, or a complete loss expression, so these hyperparameters and any invented physics loss are omitted.

Key Experimental Results¶

Main Results¶

The real-world test set contains 75 videos covering manipulation, support removal, collisions, articulated interactions, shadows, and reflections. The human study in Table 1 has 25 participants evaluating 5 randomly sampled scenarios each, producing 125 seven-way selections; VOID receives 64.8% versus Runway's 18.4%, a difference of 46.4 percentage points. This is neither a pairwise win rate nor a per-video physical success rate across the complete set.

The following entries are from Table 3: the synthetic benchmark contains 40 videos, comprising 10 traditional object/shadow/reflection cases and 30 held-out dynamic Kubric and HUMOTO cases. Metrics compare with ground-truth counterfactuals; the VLM judge is Gemini 3 Pro with access to the target video, scoring six criteria from 0 to 5 for a total of 30.

Method	PSNR ↑	LPIPS ↓	DreamSim ↓	DINOv2 ↑	FVD ↓	VLM Total ↑
ProPainter	30.48	0.10	0.10	0.89	471.13	21.38
Gen-Omnimatte	29.44	0.12	0.12	0.87	437.88	20.40
Runway	26.68	0.11	0.15	0.85	442.76	21.67
VOID	31.49	0.12	0.07	0.92	260.31	25.10

Comparisons use each model's native interface: binary masks for inpainting baselines, trimasks for Generative Omnimatte, and text explicitly describing both the removed object and expected consequences for Runway. VOID does not lead on every metric: its LPIPS of 0.12 is worse than ProPainter's 0.10, and framewise distances can penalize physically plausible falling motion with slightly displaced positions.

Ablation Study¶

Table 4 evaluates all variants on the same 75 real videos using Gemini 3 Pro; Interaction & Physics has a maximum of 5 and the total score a maximum of 30.

Config	Training Size	Interaction & Physics ↑	Total ↑	Interpretation
Kubric-Only	1200	2.63	20.36	Rigid-body data only
HUMOTO-Only	1200	2.50	20.12	Human-interaction data only
Both Datasets	1200	3.04	21.93	Equal contributions from both
Gen-Omni. Mask	Full	3.30	23.39	Black target, light gray elsewhere
VOID	Full	3.66	26.13	Full data and detailed masking pipeline

Key Findings¶

At a fixed training size of 1200, mixed data score 21.93 versus 20.36 and 20.12, supporting complementary coverage rather than simply a larger sample count.
The coarse-mask ablation drops from 26.13 to 23.39, but changes mask detail and removes the VLM masking pipeline together; the 2.74-point gap cannot be attributed entirely to the fourth color.
All three real-video judges in Table 2 rank VOID first overall, but Qwen3.5-32B scores Gen-Omnimatte at 23.44 and Runway at 22.79, so baseline rankings are not identical across judges.
The main text points to an appendix for the second-pass ablation; its table is absent from this cache, preventing a verified claim about its independent quantitative gain.

Highlights & Insights¶

An editing mask should cover causal consequences, not only pixels occupied by the removal target. Including both original and new positions addresses obsolete trajectory remnants and creates space for the new motion.
The second pass constrains generation using predicted counterfactual motion rather than preserving input motion. This ordering is useful when an edit must rewrite object trajectories.
Simulation supplies paired counterfactuals while pretrained models contribute appearance and world knowledge. Unseen-effect examples in Figure 6 suggest complementarity, but qualitative examples alone do not establish general physical reasoning.

Limitations & Future Work¶

The authors acknowledge domain gaps for unusual camera angles and close-up viewpoints, generation lasting only a few seconds, and room for higher resolution.
Real videos lack counterfactual ground truth, so conclusions rely on human preference and VLM judgment; related VLMs participating in mask generation and evaluation may introduce correlated bias, which additional judges and human evaluation mitigate but do not eliminate.
The method depends on locating all affected objects and future regions; omissions constrain subsequent edits, and erroneous first-pass flow may persist in refinement. These are mechanism-based risks, not measured failure rates reported by the paper.
Future evaluation could isolate four-value encoding, region precision, and VLM reasoning, and test longer videos, real paired data, and broader viewpoint distributions.

vs Generative Omnimatte: It supplies initialization and the object-effect decomposition foundation; VOID adds counterfactual supervision requiring other objects' motion to change, rather than merely adding a mask color.
vs ProPainter / DiffuEraser / ROSE: These methods emphasize inpainting or photometric effect removal, while VOID extends the requirement to consequences of collisions, support, and manipulation.
vs Runway Aleph: Text can state the desired outcome directly, while VOID grounds affected areas in per-frame masks; the comparison already provides Runway with expected consequences, not merely an instruction to remove the object.
vs Go-with-the-Flow: VOID reuses motion-correlated noise but derives its motion condition from first-pass counterfactual predictions to stabilize appearance after deletion.

Rating¶

Novelty: 4/5. Extends object removal to dynamic interactions and connects counterfactual data, spatial reasoning, and motion stabilization.
Experimental Thoroughness: 4/5. Includes real, synthetic, human, and multi-judge evaluations, but the human study is limited and independent component attribution remains incomplete.
Writing Quality: 4/5. Clearly explains quadmask ambiguities, while some training details and second-pass evidence depend on an unavailable appendix.
Value: 4/5. Offers a reusable direction for physics-aware video editing, but not a long-video simulator with physical guarantees.