BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

Conference: ECCV 2026
Paper: ECCV Official
Project Page: BeyondMasks Project Page
Area: Image Generation
Keywords: video object removal, causal physical consistency, paired video evaluation, vision-language model, secondary after-effects

TL;DR

Addressing the limitation of conventional video object removal protocols that evaluate only local inpainting fidelity while ignoring physical side effects like shadows and reflections, BeyondMasks introduces a temporally aligned paired benchmark (180 sequences) and a structured VLM-based evaluation protocol CORE that reveals a systematic gap between pixel-level fidelity and true causal scene restoration.

Background & Motivation

With rapid advances in large-scale generative video foundation models, video object removal has achieved remarkable perceptual realism and is increasingly integrated into open-world reasoning and instruction-driven editing pipelines. However, evaluation methodologies have remained tethered to the local inpainting paradigm. Standard benchmarks such as DAVIS and YouTube-VOS lack aligned clean background references, restricting evaluation to reconstruction fidelity strictly within the masked bounding region and treating removal as local pixel hole-filling.

In real-world scenes, physical objects are not isolated clusters of pixels; they exist within intricate spatial and causal interactions with their environment. As highlighted across real scenarios, objects cast shadows, modify local and ambient illumination, introduce specular reflections on mirrors or glossy surfaces, scatter light through translucent media, and produce irreversible dynamic traces like footprints or water ripples. Merely deleting the object's foreground pixels while leaving its induced physical after-effects yields scenes that remain causally and physically inconsistent. Existing benchmarks either synthesize clean pairs via crude alpha compositing without secondary interactions (such as HQVI), or restrict themselves to isolated photometric effects in synthetic 3D renderings (such as ROSE-Bench), lacking dynamic real-world variability and support for instruction-driven editing.

To resolve this fundamental tension, this paper reformulates video object removal as a counterfactual causal intervention problem: disentangling the target object from the generative process of the scene and undoing all induced environmental perturbations over time. The core idea is to establish BeyondMasks, a temporally aligned paired benchmark of 180 synthetic and real-world sequences where masks strictly cover only object geometry, coupled with CORE, a structured VLM-as-a-judge protocol that leverages aligned clean references to orthogonally evaluate object disappearance and secondary after-effect elimination.

Method

Overall Architecture

The methodology operates across three cohesive stages: mathematical formulation of causal intervention, construction of the paired benchmark, and decoupled VLM-based evaluation via CORE.

Formally, let \(S_t\) denote the latent physical state of a scene at time \(t\), and let \(O\) denote the presence of the target object. The observed video frame is governed by a rendering process \(V_t = \mathcal{R}(S_t, O)\). Under a counterfactual causal intervention where the object is removed (\(O = \emptyset\)), the scene renders to the clean background state \(V_t^{\mathrm{bg}} = \mathcal{R}(S_t, \emptyset)\). Because \(O\) alters \(S_t\) via indirect environmental interactions (shadows, reflections, volumetric scattering), true object removal requires approximating the interventional distribution \(\mathcal{R}(S_t, \emptyset)\) rather than hallucinating plausible textures inside a localized mask.

BeyondMasks implements this formulation by curating temporally aligned paired sequences across synthetic and real-world scenes with a specialized "object-only" masking policy. Subsequently, CORE ingests three aligned video streams to conduct comparative multimodal reasoning and output decoupled evaluation scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Paired Data Creation<br/>90 Veo 3 Synthetic + 90 Tripod Real Sequences"] --> B["Causal Intervention & Annotation Strategy<br/>SAM 2 + CoTracker 3 Object-Only Masks"]
    B --> C["Taxonomy of Causal After-Effects<br/>Shadow / Reflection / Translucent / Steam / Light / Causal Traces"]
    C --> D["CORE Structured VLM Protocol<br/>Tri-Stream Aligned Comparative Reasoning (Gemini 3.1)"]
    D --> E["Decoupled Causal Consistency Scoring<br/>ObjectScore (CORE-OS) + AfterEffectScore (CORE-AES)"]

Key Designs

1. Causal Intervention & Annotation Strategy: Unmasked Secondary Effects Force Physical Reasoning

Existing video removal benchmarks suffer from a lack of paired clean reference videos where both the object and its secondary effects are absent. BeyondMasks remedies this by assembling 180 temporally aligned paired video sequences (average duration \(\sim 5.5\) seconds). The synthetic subset (90 sequences) leverages Veo 3 by first generating a background-only reference \(V^{\mathrm{bg}}\) and subsequently introducing the target object along with realistic physical interactions via controlled editing to yield \(V^{\mathrm{obj}}\). The real-world subset (90 sequences) uses a tripod-stabilized camera setup with locked exposure and focus parameters to record back-to-back video pairs before and after physically removing the object.

Crucially, in the annotation pipeline combining SAM 2 with CoTracker 3 point tracking, annotators strictly segment only the physical boundaries of the object itself, deliberately leaving all induced physical effects (cast shadows, reflections, illumination changes) unmasked. This deliberate policy deprives models of the spatial shortcut of treating removal as an enlarged inpainting task, forcing them to reason about environmental physical consistency.

2. Taxonomy of Causal After-Effects: Comprehensive Coverage of Physical Mechanisms

Rather than categorizing scenes by semantic object classes, BeyondMasks establishes a mechanism-driven taxonomy covering six distinct physical interaction modalities: - Shadow (127 instances): Occlusion of direct illumination; removal requires recovering the true unoccluded radiance field. - Reflection (61 instances): Specular or diffuse reflections on mirrors, glass, or water bodies; removal demands eliminating both direct and reflected evidence while restoring underlying surface texture. - Translucent effects (46 instances): Refraction and attenuation through semi-transparent materials; removal requires recovering background structures partially visible through the medium. - Steam/scattering (20 instances): Time-varying, spatially diffuse volumetric media such as smoke and steam; removal requires restoring spatio-temporal background continuity. - Light source effects (38 instances): Active illumination emitted by the object; removal requires compensating for local and ambient overexposure and shading shifts. - Causal physical effects (37 instances): Non-attached geometric or material disturbances induced by object motion or contact, including water ripples, displaced dust, footprints, and deformed surfaces.

A control group with minimal secondary effects is also provided, and all samples include validated instruction prompts to support both mask-guided and instruction-driven editing pipelines.

3. CORE Structured VLM Protocol: Decoupled Multi-View Evaluation with Aligned References

Standard pixel metrics (PSNR, SSIM, LPIPS) cannot verify whether an algorithm has satisfied counterfactual physical consistency, frequently assigning similar distance scores to outputs with vastly different physical side-effect artifacts. To address this, CORE adopts Gemini 3.1 Pro as an automated judge. It ingests three temporally aligned video sequences: the input object-present video \(V^{\mathrm{input}}\), the clean reference background \(V^{\mathrm{gt}}\), and the edited result \(V^{\mathrm{result}}\).

Operating under a structured prompt, the VLM executes four sequential reasoning steps: 1. Locate: Identify the target entity and its spatial-temporal after-effects in \(V^{\mathrm{input}}\); 2. Determine: Establish the ground-truth appearance from the clean background reference \(V^{\mathrm{gt}}\); 3. Compare: Evaluate \(V^{\mathrm{result}}\) against \(V^{\mathrm{gt}}\) for physical plausibility and naturalness; 4. Categorize & Score: Disentangle observed discrepancies into entity-related vs. after-effect-related failures.

CORE produces two distinct 1-to-5 ratings: - ObjectScore (CORE-OS): Measures the completeness of object disappearance and background restoration plausibility; - AfterEffectScore (CORE-AES): Isolates the thoroughness of eliminating secondary physical after-effects (shadows, reflections, dynamic traces).

In human validation across 8 annotators, 5 models, and 20 video pairs, CORE achieves a Pearson correlation of 0.615 for CORE-OS (human inter-rater correlation is 0.693) and 0.733 for CORE-AES (human inter-rater correlation is 0.785), approaching the human consensus ceiling.

Loss & Training

As a benchmark and diagnostic evaluation framework, BeyondMasks introduces no specialized loss function. All evaluated baseline models are run using their official open-source checkpoints and default inference settings at native resolution: - Mask-based methods receive SAM 2 + CoTracker 3 object-only binary masks without dilated hulls; - Text-driven instruction editing methods receive standardized prompts (e.g., "Remove the [object]").

Key Experimental Results

Main Results

The authors benchmarked nine competitive models across text-guided (T), mask-guided (M), and hybrid (T+M) paradigms. The comparative performance across pixel fidelity, temporal realism, and CORE metrics is detailed below:

Method	Type	PSNR ↑	SSIM ↑	LPIPS ↓	FVD ↓	CORE-OS ↑	CORE-AES ↑
Lucy Edit (arXiv 2025)	T	18.5884	0.7239	0.2758	480.08	1.443	1.551
VACE (ICCV 2025)	T+M	19.9932	0.7856	0.2077	418.79	1.799	1.698
CoCoCo (AAAI 2025)	T+M	20.8565	0.7287	0.2404	381.84	1.986	2.043
Gen. Omnimatte (CVPR 2025)	T+M	23.8106	0.8175	0.1968	250.76	3.771	3.021
ProPainter (ICCV 2023)	M	23.5979	0.8429	0.1635	225.28	3.063	2.368
DiffuEraser (arXiv 2025)	M	25.1880	0.8769	0.1124	164.26	3.494	2.625
MiniMax (NeurIPS 2025)	M	23.1954	0.8218	0.1682	203.64	3.631	2.553
ROSE (NeurIPS 2025)	M	23.8062	0.8173	0.1659	316.04	3.784	2.920
OmnimatteZero (SIGGRAPH 2025)	M	22.9306	0.8127	0.1638	227.24	3.190	2.499

Ablation Study / Breakdown Across After-Effect Categories

To evaluate how models cope with distinct physical mechanisms, the performance averaged across the top-6 performing models is broken down across the six after-effect categories:

After-Effect Category	Sample Count	PSNR ↑	SSIM ↑	LPIPS ↓	CORE-OS ↑	CORE-AES ↑	Physical Restoration Bottleneck
Shadow	127	23.6870	0.8298	0.1621	3.498	2.520	Unmasked diffuse penumbras mistaken for static background texture
Reflection	61	23.9842	0.8558	0.1580	3.332	2.446	Mirror symmetry and specular highlights persist across frames
Light source	38	22.4233	0.8201	0.1849	3.554	2.248	Inability to adjust global and local ambient exposure fields
Steam	20	22.8375	0.8466	0.1624	3.236	2.032	Diffuse spatio-temporal boundary induces severe residual blur
Translucent	46	22.9326	0.8156	0.2162	3.256	2.701	Mask zeroing discards visible background geometry behind medium
Causal	37	22.7912	0.8163	0.1867	3.486	2.141	Irreversible dynamic deformations (footprints, ripples) completely ignored

Key Findings

• Severe Disconnect Between Pixel Similarity and Causal Correctness: DiffuEraser achieves the strongest pixel fidelity across PSNR (25.1880), SSIM (0.8769), and LPIPS (0.1124), yet its CORE-AES is only 2.625, substantially behind ROSE (2.920) and Generative Omnimatte (3.021). Pixel metrics reward conservative local background patching while failing to penalize unmasked physical errors.
• Entity Removal vs. After-Effect Elimination: Across all top mask-guided methods, CORE-OS remains relatively high (3.0 to 3.8), indicating that identifying and inpainting the object body is largely solved. However, CORE-AES remains severely depressed (2.0 to 3.0), establishing that secondary after-effects are the dominant failure mode.
• Dynamic Traces and Volumetric Media are the Hardest: Models perform worst on Steam (CORE-AES 2.032) and dynamic Causal effects (CORE-AES 2.141). Existing video inpainting architectures lack temporal physical causality models to revert disturbed water or foot-impressed sand back to their pristine states.
• The Context-Loss Pitfall in Mask-Based Pipelines: Preprocessing pipelines that zero out masked regions discard valuable partial observations visible through translucent or refractive objects, forcing generative backbones to hallucinate structures that were partially visible in the original frames.

Highlights & Insights

• Reframing Video Removal as Counterfactual Intervention: Grounding evaluation in causal recovery of \(\mathcal{R}(S_t, \emptyset)\) formally separates visual plausibility from physical correctness, providing a principled paradigm for video editing benchmarks.
• Strict Object-Only Annotation Design: Denying models the crutch of manually expanded masks directly diagnoses their intrinsic capacity for multi-modal physical commonsense and optical reasoning.
• Conditioned Multi-Stream VLM-as-a-Judge: Providing the VLM with aligned ground truth transforms subjective aesthetic grading into rigorous counterfactual verification, setting a high standard of human-judge correlation.

Limitations & Future Work

• Benchmark Scale: Comprising 180 curated sequences, BeyondMasks serves as an intensive diagnostic test suite rather than a large-scale pretraining corpus, constrained by the manual effort required for tripod capture and verification.
• Judge Model Dependency: CORE currently relies on proprietary frontier models (Gemini 3.1 Pro); domain shifts or subtle rendering artifacts could occasionally trigger localized vision-language hallucinations.
• Future Directions: Developing physics-informed video inpainting networks that explicitly predict illumination fields, surface normals, and material transport to undo secondary interactions rather than relying purely on diffusion feature matching.

Related Work & Insights

• vs DAVIS / YouTube-VOS: Standard VOS datasets lack aligned background ground truth, limiting evaluation strictly to masked-area fidelity; BeyondMasks supplies 180 clean background reference pairs to measure full-scene recovery.
• vs HQVI: HQVI builds paired data via alpha-compositing without physical environmental interactions; BeyondMasks captures authentic and simulated physical coupling including shadows, specular reflections, and dynamic disturbances.
• vs ROSE-Bench: ROSE-Bench contains only 60 synthetic rendered sequences focused on static photometric effects; BeyondMasks doubles the scale (180 sequences), incorporates real tripod captures, models dynamic physical traces (water ripples, footprints), and supports text-instruction evaluation.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates video object removal as a causal physical intervention problem and introduces an unmasked-effect evaluation paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across 9 methods, 3 input modalities, 6 physical effect categories, and validated human agreement studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Precise mathematical formulation, clean taxonomical breakdown, and rigorous prose structure.
• Value: ⭐⭐⭐⭐⭐ Exposes a critical blind spot in current generative video metrics and establishes an indispensable benchmark for physically faithful video editing.