InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Video Object Insertion, 4D Scene Reconstruction, Scene Flow Tracking, Optics-Aware Representation Alignment, Video Diffusion Models

TL;DR

InsertAnywhere couples 4D scene reconstruction and KNN scene flow tracking with a stop-gradient optics-aware feature alignment strategy, enabling production-grade video object insertion with realistic cast shadows and reflections under dynamic occlusions from a single-frame 3D placement.

Background & Motivation

Video Object Insertion (VOI) aims to seamlessly embed a novel reference object into an existing video sequence. It plays a pivotal role in visual effects, commercial advertising, and digital content creation. To achieve production-grade quality, the inserted object must maintain rigorous spatial, temporal, and photometric consistency with the surrounding environment across camera motion, dynamic occlusions, and changing scene illumination. However, current generative video editing approaches face a fundamental dilemma between geometric controllability and physical optical synthesis.

Existing methods largely fall into two paradigms. Mask-free generative models typically guide insertion via text prompts or an edited anchor frame, relying on the generative prior of diffusion transformers. While their unconstrained generation area theoretically allows them to synthesize lighting variations such as cast shadows, the lack of strict spatial boundaries frequently degrades and distorts unedited background areas, leading to temporal drift and failure under complex foreground occlusions. In contrast, mask-conditioned models formulate insertion as localized inpainting governed by an explicit spatial mask. While this strictly preserves background integrity, the rigid mask boundary locks generation strictly within the object silhouette, completely preventing the synthesis of optical interactions—such as contact shadows, ambient occlusion, and reflections—that naturally extend beyond the boundary into the scene. Furthermore, single 2D reference images lack depth and scale cues, leaving models unable to reason about the 4D dynamics of moving support surfaces.

To reconcile spatial controllability with photometric realism, this paper proposes a two-stage decoupled framework. Users adjust the object's 3D pose, scale, and placement in an interactive 3D interface on an anchor frame. The system then propagates the object's rigid motion using spatio-temporal point clouds and local scene flow to generate an occlusion-robust mask sequence, while an optics-aware representation alignment mechanism trains the diffusion model to project realistic external lighting effects. Core idea: propagate geometrically grounded masks via 4D scene reconstruction and KNN scene flow, and employ a stop-gradient extended-mask feature alignment loss on the curated ROSE++ dataset so that a tightly bound mask model learns to synthesize realistic cast shadows and reflections beyond its boundary.

Method

Overall Architecture

InsertAnywhere consists of two synergistic stages: Geometrically Grounded Mask Propagation and Optics-Aware Video Synthesis. In the first stage, the monocular source video is reconstructed into a decoupled 4D point cloud representation separating static scene structure from dynamic elements. A 3D object point cloud generated from the reference image is placed into the scene, interactively adjusted by the user in an anchor frame, and tracked forward and backward in time via KNN scene flow aggregation before being reprojected into a temporally coherent binary mask sequence. In the second stage, a mask-conditioned video diffusion backbone is fine-tuned with LoRA on the newly constructed ROSE++ quadruplet dataset. By pairing first-frame anchoring with Optics-Aware Representation Alignment against an extended mask teacher, the model synthesizes photorealistic insertions with physically plausible shadows and highlights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Source Video V_src + Reference Image I_ref"] --> B["4D Geometrically Grounded Mask Propagation<br/>Uni4D Decoupled Reconstruction + TRELLIS 3D Anchor"]
    B --> C["KNN Scene Flow Temporal Tracking<br/>SEA-RAFT 3D Flow Aggregation + SAM 2 Reprojection"]
    C --> D["First-Frame Anchoring & ROSE++ Fine-Tuning<br/>OmniPaint First Frame + De-lit Reference Images"]
    D --> E["Optics-Aware Representation Alignment<br/>Extended-Mask Teacher Feature Alignment L_align"]
    E --> F["Output: Production-Grade Seamless Video"]

Key Designs

1. 4D Geometrically Grounded Mask Propagation: Decoupled Point Clouds and KNN Scene Flow Tracking

Standard 2D tracking or planar homographies degrade severely when an object is placed on a non-rigidly moving support (e.g., a rolling luggage cart) or passes behind complex dynamic occluders. To provide robust spatial constraints, the framework first reconstructs the monocular video into a 4D spatio-temporal point cloud using Uni4D, integrating depth estimation (UniDepthV2), optical flow (SEA-RAFT), and camera pose tracking to disentangle static background geometry from dynamic foreground entities. The 2D reference image \(I_{ref}\) is lifted into an initial 3D rigid point cloud \(y\) via TRELLIS. The user interacts with an anchor frame \(s\) to set the rigid transformation:

\[y_s = s_{obj} R_{obj} y + t_{obj}\]

where rotation \(R_{obj}\), translation \(t_{obj}\), and scale \(s_{obj}\) are interactively aligned with the supporting geometry. To ensure the object moves synchronously with its local physical substrate, dense 2D optical flow is estimated using SEA-RAFT and back-projected to compute 3D displacement vectors \(\Delta p_i^t\). The framework aggregates the 3D displacement vectors of the Top-\(K\) nearest scene points around the object centroid:

\[\Delta p_{obj}^t = \frac{1}{K} \sum_{k \in \text{TopK}} \Delta p_k^t\]

The object point cloud is updated iteratively as \(y'_t = y'_{t-1} + \Delta p_{obj}^{t-1}\), and then projected onto the image plane using estimated camera intrinsics \(K\) and extrinsics \(P_t = [R_t | t_t]\). Rasterized silhouettes are refined via SAM 2, yielding a temporally consistent, occlusion-aware binary mask sequence \(\{M_t\}_{t=1}^T\).

2. Optics-Aware Representation Alignment: Extended-Mask Teacher Guidance via Stop-Gradient

Conventional mask-conditioned video diffusion models (e.g., VACE) strictly confine generation within the input mask to preserve background pixels, cutting off natural shadows, specular highlights, and ambient reflections. To overcome this limitation during inference while maintaining a tight, user-friendly input mask, the paper introduces Optics-Aware Representation Alignment.

On paired training data, an optics-aware extended mask \(M_{ext}\) is derived by thresholding the absolute pixel differences between the object-free source video \(V_{src}\) and the object-present target video \(V_{tgt}\), capturing both the object and its photometric footprint (\(M \subset M_{ext}\)). During training, both the primary mask \(M\) and the extended mask \(M_{ext}\) are tokenized and processed through the shared diffusion transformer backbone. The model aligns intermediate representations \(F^l\) from the fine-mask branch with \(F^l_{ext}\) from the extended-mask branch across multiple transformer blocks:

\[\mathcal{L}_{align} = \sum_{l} \| F^l - \text{sg}(F^l_{ext}) \|_2^2\]

where \(\text{sg}(\cdot)\) denotes the stop-gradient operator. The extended-mask branch serves as a fixed optical teacher, forcing the fine-mask branch to anticipate and activate the surrounding latent representation. Consequently, during inference, the network synthesizes realistic external shadows and reflections even when conditioned only on the tightly bound object mask.

3. ROSE++ Dataset Construction and First-Frame Anchoring: Mitigating Copy-and-Paste Shortcuts

Supervised training of optical interactions requires quadruplets of source video, target video, object mask sequence, and isolated reference image. However, directly cropping reference objects from target videos in existing benchmarks (like ROSE) introduces severe shortcuts: the model memorizes fixed lighting and viewing angles, degenerating into a trivial copy-and-paste mechanism.

To address this, the authors construct ROSE++. Candidate reference images are generated on neutral white backgrounds using an image editing model to remove scene-dependent illumination. A DINOv2 feature extractor \(\Phi_{DINO}\) computes the cosine similarity between generated candidates \(\hat{o}_k\) and multi-view crops \(\{f_j\}_{j=1}^N\) extracted from target frames:

\[s_k = \frac{1}{N} \sum_{j=1}^N \text{sim}(\Phi_{DINO}(\hat{o}_k), \Phi_{DINO}(f_j))\]

The candidate with the highest similarity score is chosen through rejection sampling. Furthermore, the synthesis pipeline leverages an image-level insertion model (OmniPaint) to generate an edited first frame as an explicit conditioning anchor. This injects robust object identity and high-frequency textural details that temporal attention layers naturally propagate across subsequent frames.

Loss & Training

The framework fine-tunes the pre-trained VACE video diffusion model using LoRA adapters. The total objective optimizes the diffusion noise prediction loss \(\mathcal{L}_{diff}\) alongside the optics-aware alignment loss \(\mathcal{L}_{align}\):

\[\mathcal{L}_{total} = \mathcal{L}_{diff} + \lambda_{align} \mathcal{L}_{align}\]

The VAE encoder/decoder and text tokenizers remain frozen, while LoRA parameters in the attention blocks are updated. The stop-gradient operator on the extended-mask branch guarantees training stability without degenerate feature collapse.

Key Experimental Results

Main Results

The model is evaluated on VOIBench, a comprehensive benchmark of 200 video sequences spanning indoor, outdoor, and natural environments under diverse lighting conditions and dynamic occlusions. Metrics include Subject Consistency (CLIP-I, DINO-I) and VBench video quality dimensions (Background Consistency, Subject Consistency, Motion Smoothness, Imaging Quality).

Method	Type	CLIP-I ↑	DINO-I ↑	Background Consistency ↑	Subject Consistency ↑	Motion Smoothness ↑	Imaging Quality ↑
Pika-Pro	Commercial	0.4940	0.3856	0.9080	0.8720	0.9889	0.6546
Kling	Commercial	0.6349	0.5028	0.9335	0.9494	0.9940	0.7069
AnyV2V	Open Source	0.7033	0.2217	0.8884	0.8699	0.9795	0.5973
ReVideo	Open Source	0.7385	0.3651	0.9391	0.9403	0.9906	0.6526
Señorita	Open Source	0.7499	0.3982	0.9262	0.9266	0.9902	0.6333
VACE (with our mask)	Open Source	0.7368	0.5060	0.9011	0.8855	0.9887	0.6046
InsertAnywhere (Ours)	Full Model	0.8132	0.5669	0.9503	0.9534	0.9925	0.7473

Note: Data from Table 1 of the original paper. Except for Motion Smoothness where it is virtually tied with Kling within per-clip noise, InsertAnywhere outperforms all commercial and academic baselines across all metrics.

Ablation Study

The ablation study validates each core component cumulatively (Table 3): Config (a) employs only the 4D-aware mask; Config (b) adds first-frame inpainting anchoring; Config (c) adds ROSE++ LoRA fine-tuning; Ours incorporates the Optics-Aware Representation Alignment loss \(\mathcal{L}_{align}\).

Config	Key Modification	CLIP-I ↑	DINO-I ↑	Background Consistency ↑	Imaging Quality ↑	Multi-View Consistency ↑	Note
Config (a)	4D Mask Only	0.7532	0.3861	0.9232	0.6298	0.5308	Resolves occlusion failures, but low identity fidelity
Config (b)	+ First-Frame Anchoring	0.7880	0.5135	0.9175	0.6318	0.5436	Substantially improves subject identity preservation
Config (c)	+ ROSE++ LoRA Tuning	0.8122	0.5678	0.9429	0.7101	0.5857	Stabilizes temporal consistency and texture
Ours	+ Alignment Loss \(\mathcal{L}_{align}\)	0.8132	0.5669	0.9503	0.7473	0.5865	Synthesizes realistic cast shadows and reflections

Note: Data from Table 3 of the original paper.

Ablation on shadow and optics rendering quality evaluated on a 25-sample held-out split of ROSE++ (Table 4 Right): - VACE (with our mask): LPIPS 0.0130, PSNR 19.8345 dB, SSIM 0.9877 on Optics Mask - Ours w/o \(\mathcal{L}_{align}\): LPIPS 0.0121, PSNR 19.9065 dB, SSIM 0.9883 on Optics Mask - Ours (Full Model): LPIPS 0.0089, PSNR 27.2362 dB, SSIM 0.9920 on Optics Mask

Key Findings

1. +7.3 dB PSNR Surge in the Optics Region: In the optical impact zone (Optics Mask), adding \(\mathcal{L}_{align}\) drives PSNR up from 19.91 dB to 27.24 dB—a notable +7.3 dB jump—while lowering LPIPS from 0.0121 to 0.0089. This confirms that representation alignment effectively teaches the diffusion transformer to cast shadows and reflections outside the input mask.
2. Preservation of Unedited Backgrounds: The unedited-region PSNR remains virtually identical across configurations (VACE: 33.27 dB, Ours w/o \(\mathcal{L}_{align}\): 33.31 dB, Ours: 33.30 dB). The optical expansion is strictly localized to the physical interaction zone without inducing global background drift.
3. Prevention of Copy-and-Paste Degradation: Comparing the full model against a baseline trained with random target video crops (Ours-random), multi-view consistency increases from 0.5295 to 0.5865 (Table 4 Left). The DINO-guided de-lit reference images force the network to infer true 3D lighting rather than memorizing surface pixels.

Highlights & Insights

• Teacher-Student Cross-Mask Feature Distillation: Employing a stop-gradient operator on an extended-mask teacher provides a clean mechanism to train a standard mask-conditioned model to break free of spatial boundary constraints during inference.
• Local KNN Scene Flow over Global Rigid Assumptions: Instead of assuming uniform scene rigidity, aggregating 3D scene flow vectors over the \(K\)-nearest neighbor points allows the object to synchronize dynamically with moving substrates (such as carts or vehicles).
• Curated ROSE++ Benchmark: By constructing neutral reference images via DINO rejection sampling, the authors turn an object removal benchmark into a clean supervised video insertion quadruplet dataset.

Limitations & Future Work

• Absence of Soft-Body Physics and Non-Rigid Contact: The rigid 3D point cloud formulation cannot simulate non-rigid deformations, such as an object depressing a soft mattress or a heavy box sinking into snow.
• Sensitivity to Upstream Monocular Reconstruction: The pipeline relies on monocular depth and camera pose estimators; in low-texture, reflective, or murky aquatic environments, noisy scene flow can lead to mask edge jitter.

Related Work & Insights

• vs Mask-Free Video Insertion (OmniInsert, AnyV2V, Kling): Mask-free methods allow unconstrained optical hallucination but suffer from catastrophic background distortion and lack fine-grained 3D placement control. InsertAnywhere provides precise 3D spatial grounding while retaining background integrity.
• vs Mask-Conditioned Video Inpainting (VACE, VideoAnyDoor, DreamInsert): Conventional inpainting frameworks lock generation inside the mask boundary, creating a floating "sticker" look. InsertAnywhere's representation alignment enables boundary-overflowing light interactions without modifying test-time mask inputs.

Rating

• Novelty: ⭐⭐⭐⭐⭐ An elegant bridging of 4D geometric mask propagation and stop-gradient optical feature alignment.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across VOIBench, paired optics mask ablations, and attention map analyses.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, intuitive figures, and well-structured mathematical motivation.
• Value: ⭐⭐⭐⭐⭐ Directly addresses key bottlenecks impeding VOI adoption in film VFX and commercial video production.

Related Papers

• [ECCV 2026] Unwarping the Lens: A Physics-Grounded Approach to Video Glasses Removal
• [ICML 2026] Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
• [CVPR 2026] EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing
• [ECCV 2026] GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoising
• [ECCV 2026] DTI: Dynamic Trajectory Initialization for Generative Face Video Super-Resolution