Skip to content

SceneDiff: A Benchmark and Method for Multiview Object Change Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: https://yuqunw.github.io/SceneDiff
Area: 3D Vision
Keywords: Object Change Detection, Multiview Perception, 3D Scene Understanding, Robotics

TL;DR

Addressing object change detection across non-overlapping camera trajectories, this paper introduces the SceneDiff Benchmark with 350 video pairs and dense instance annotations, alongside a training-free framework combining 3D geometry co-registration and region-level feature comparison.

Background & Motivation

Object change detectionโ€”identifying objects that have been added, removed, or moved across captures of a scene taken at different timesโ€”serves as a core touchstone of spatial intelligence. This capability is pivotal for downstream applications such as robotic room tidying, construction tracking, warehouse logistics verification, and post-disaster damage cataloging. However, real-world data collection rarely follows identical paths; when before-and-after video trajectories diverge, dramatic viewpoint alterations, perspective distortion, field-of-view departures, occlusions, and illumination variations frequently generate apparent discrepancies, leading traditional image differencing or two-view matching pipelines to trigger pervasive false positives.

The longstanding bottleneck in this domain stems from the lack of realistic multiview object-level benchmarks and the limitations of existing methodological paradigms. Prior datasets either enforce near-identical camera viewpoints or offer merely pixel-level semantic change maps on synthetic scenes or sparse image pairs, failing to assess whether an algorithm can consistently track the same physical object across arbitrary video trajectories. On the algorithmic side, synthesis-and-comparison strategies based on NeRF or 3D Gaussian Splatting (3DGS) require dense, omnidirectional views for clean rendering and degrade catastrophically under wide-baseline novel views or sparse passes, while task-specific fine-tuned 2D/3D networks suffer from severe sim-to-real transfer gaps.

To decouple true physical changes from perspective shifts and occlusions, this work extends the philosophy of textual diff tools ("align first, then inspect differences") to 3D space: first utilizing invariant background geometry to co-register before-and-after video streams into a unified coordinate frame, and then lifting comparisons from noisy pixel/point clouds to coherent object-level instances. Core idea: leverage a feed-forward 3D geometry foundation model for joint camera registration and co-visibility frame pairing, construct an asymmetric visibility mask to discard occlusion artifacts, and combine depth reprojection, DINOv3 appearance reprojection, and SAM region-pooled matching to achieve robust training-free multiview object change detection and temporal association.

Method

Overall Architecture

SceneDiff takes two video sequences of a scene captured before (\(I_{\text{pre}}\)) and after (\(I_{\text{post}}\)) changes as input. It outputs the set of physically changed objects, their 2D segmentation masks across all visible frames, and their change status (Added, Removed, or Moved). The algorithm proceeds through three interconnected phases: first, a feed-forward 3D vision geometry model jointly estimates camera parameters and dense depth maps to select pairs of frames with high co-visibility; second, asymmetric geometry reprojection, dense appearance feature reprojection, and geometry-free region matching are computed for each selected pair and pooled over SAM candidate masks; finally, 3D voxelization reprojection filtering, entropy thresholding, temporal instance clustering, and greedy bipartite matching produce persistent 3D/2D object tracks and classify change categories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Before and After Video Sequences<br/>I_pre and I_post"] --> B["Joint Geometry Regression and Frame Pairing<br/>ฯ€3 joint pose/depth + 50% co-visibility pairing"]
    B --> C["Asymmetric Geometry Comparison<br/>Unidirectional depth reprojection + M_pre mask"]
    B --> D["Reprojected Appearance Comparison<br/>DINOv3 cosine dissimilarity + mask filtering"]
    B --> E["Region-Level Appearance Matching<br/>SAM mask mean-pooling + global search"]
    C & D & E --> F["Unified Region Change Scoring and Aggregation<br/>Weighted sum + 3D voxel back-projection filtering"]
    F --> G["Temporal Instance Association and Classification<br/>ConceptGraph-style clustering + bipartite matching"]
    G --> H["Output Changed Object Instances and Types<br/>Added / Removed / Moved"]

Key Designs

1. Joint Geometry Regression and Frame Pairing: Establishing Reliable Correspondences Across Unconstrained Trajectories Arbitrary handheld or egocentric video trajectories exhibit unpredictable viewpoint deviations and speed changes, rendering naive sequential frame matching futile. SceneDiff feeds both \(I_{\text{pre}}\) and \(I_{\text{post}}\) jointly through the feed-forward visual geometry foundation model \(\pi^3\), directly estimating depth maps, intrinsic matrices, and extrinsic camera poses \(\{D_n, T_n, K_n\}\) in a shared coordinate frame, normalized to fit within a canonical \([-1, 1]^3\) bounding volume. To filter out uninformative view pairs, the system measures bidirectional co-visibilityโ€”the average fraction of mutually visible reprojected pixels between views. For each frame \(I_n^{\text{pre}}\), it matches after-frames \(I_{n'}^{\text{post}}\) where co-visibility exceeds \(50\%\) (or selects the frame with highest co-visibility), guaranteeing that subsequent change detection is computed over overlapping visual fields.

2. Asymmetric Geometry and Reprojected Appearance Comparison: Resolving Occlusion-Induced Discrepancy Perspective shifts create depth mismatches that do not correspond to actual object changes. When reprojecting pixels from \(I_{\text{pre}}\) to \(I_{\text{post}}\), an object existing strictly in \(I_{\text{pre}}\) (a removed object) yields a predicted depth that is strictly smaller than the observed background depth in \(I_{\text{post}}\), giving a consistent positive depth difference. Conversely, negative depth differences frequently stem from foreground occlusions or objects existing solely in \(I_{\text{post}}\). Capitalizing on this optical asymmetry, SceneDiff defines a directional visibility mask: $\(M_{\text{pre}}(p) = \left( E_{\text{geom}}(p) \ge \tau_{\text{occ}} \right) \land V_{\text{pre}\to\text{post}}(p)\)$ where \(E_{\text{geom}}(p) = D_{\text{post}}(p') - D_{\text{post}}^*(p')\) is the depth difference, \(\tau_{\text{occ}} = -0.02\), and \(V_{\text{pre}\to\text{post}}\) is the geometric visibility mask. Across non-occluded regions permitted by \(M_{\text{pre}}\), the method extracts dense DINOv3 features and computes the reprojected cosine distance \(E_{\text{feat}}(p) = M_{\text{pre}}(p) (1 - \cos(F_{\text{pre}}(p), F_{\text{post}}(p')))\), providing strong robustness against lighting and micro-viewpoint fluctuations.

3. Region-Level Appearance Matching and Unified Scoring: Bridging Noisy Pixels and Object Boundaries Pixel-wise reprojections can be brittle to minor calibration errors, depth boundary bleeding, or significant spatial displacements where geometric reprojection misses the counterpart entirely. SceneDiff deploys SAM to segment candidate regions \(\mathcal{R}\) in all images and aggregates features over each region \(r\). In parallel with reprojection scores, it evaluates geometry-free region matching \(E_{\text{region}}(r)\): mean-pooling DINOv3 features within region \(r\) (\(\bar{F}_{\text{pre}}^r\)) and searching across all candidate masks in \(I_{\text{post}}\) for the best cosine match: $\(E_{\text{region}}(r) = 1 - \max_{s \in \mathcal{R}_{\text{post}}} \cos\left(\bar{F}_{\text{pre}}^r, \bar{F}_{\text{post}}^s\right)\)$ If a translated object finds an exact visual match elsewhere in the scene, \(E_{\text{region}}\) suppresses false alarms. All cues are pooled over region masks and combined linearly: \(\Delta^{\text{pre}}(r) = \lambda_{\text{geom}} \bar{E}_{\text{geom}}(r) + \lambda_{\text{feat}} \bar{E}_{\text{feat}}(r) + \lambda_{\text{region}} E_{\text{region}}(r)\), with weights set to \(1.0\), \(0.5\), and \(0.1\). Swapping the image pair yields the counterpart map \(\Delta^{\text{post}}\).

4. 3D Voxel Back-Projection Filtering and Video-Level Instance Association: Unifying Object Identity Across Frames Per-frame predictions often suffer from transient specular glints, noise, and intra-sequence fragmentation. To achieve cross-view coherence, SceneDiff unprojects all 2D score maps \(\Delta_n\) into 3D space, computes average scores across voxels in the scene point cloud, and pools them back to the 2D masks before applying maximum entropy thresholding. To associate regions into distinct physical instances across the video, the method adapts ConceptGraph's incremental clustering: candidate regions are merged into existing 3D object instances if their combined feature cosine similarity and 3D Euclidean point overlap score \(S(r) = S_{\text{feat}}(o, r) S_{\text{geo}}(o, r)\) surpasses \(\sigma_{\text{merge}} = 1.4\). Finally, greedy bipartite matching between the consolidated object clusters \(O_{\text{pre}}\) and \(O_{\text{post}}\) with a threshold of \(\tau_{\text{sim}} = 0.7\) labels matched objects as Moved, while unmatched elements are categorized as Removed (if in \(O_{\text{pre}}\)) or Added (if in \(O_{\text{post}}\)).

Key Experimental Results

Main Results

On the proposed SceneDiff Benchmark (comprising the diverse real-world subset SD-V and the egocentric kitchen subset SD-K), SceneDiff is evaluated against pixel-level baselines (MV3DCD), bounding-box detectors (CYWS-2D, CYWS-3D), 3D Gaussian Splatting (3DGS-CD), and an MLLM baseline (Qwen2.5-VL + SAM3), as shown below (from Table 2 of the paper):

Subset Method Output Type px/im IoU (%) obj/im AP (%) obj/sc AP (%)
SD-V (Varied) MV3DCD Pixel 22.1 - -
SD-V CYWS-2D* (finetuned) Bounding Box - 24.5 -
SD-V VLM (Qwen2.5-VL + SAM3) Region Tracks 22.9 7.1 5.3
SD-V 3DGS-CD Region Tracks 14.7 1.0 0.5
SD-V SceneDiff (Ours) Region Tracks 39.1 43.8 22.8
SD-K (Kitchen) MV3DCD Pixel 9.1 - -
SD-K CYWS-2D* (finetuned) Bounding Box - 16.8 -
SD-K VLM (Qwen2.5-VL + SAM3) Region Tracks 12.4 3.6 2.1
SD-K 3DGS-CD Region Tracks 4.8 0.1 0.1
SD-K SceneDiff (Ours) Region Tracks 20.8 20.9 10.6

On the established two-view RC-3D benchmark (Table 5 in the paper), SceneDiff delivers substantial gains in bounding box AP50:

Method Input Modality Both AP50 (%) Present AP50 (%) Absent AP50 (%)
CYWS-2D RGB 14.0 - -
CYWS-3D RGB (monocular depth) 41.0 - -
CYWS-3D* RGB + Ground-Truth Sensor Depth 50.0 - -
SceneDiff (Ours) RGB-only (Training-Free) 65.3 72.3 59.1

Ablation Study

Ablations on the SD-V test set investigate the impact of geometry backbones, semantic feature extractors, and individual score components (from Table 6 in the paper; values reflect differences \(\Delta\) from full \(\pi^3 + \text{DINOv3}\)):

Category Configuration px/im IoU (%) obj/im AP (%) obj/sc AP (%) Notes
Full Model \(\pi^3\) + DINOv3 39.1 43.8 22.8 Default full configuration
Geometry Model VGGT + DINOv3 -2.2 -8.1 -8.7 Reduced multi-view consistency
Geometry Model FASt3R + DINOv3 -23.7 -39.8 -22.4 Poor registration severely degrades results
Appearance Model \(\pi^3\) + DINOv2 +0.4 +1.3 -1.5 Slight drop in scene-level cross-view tracking
Appearance Model \(\pi^3\) + DINOv1 -3.0 +1.5 -2.7 Weaker invariant semantics
Feature Cues Geometry only (\(E_g\)) +2.1 -12.1 -10.3 Fine pixel edges, but object AP drops drastically
Feature Cues Reprojected Feat only (\(E_f\)) -6.1 +0.1 0.0 Dominant contributor to instance detection
Feature Cues Region Matching only (\(E_r\)) -14.8 -14.2 -0.8 Prone to false matches without geometry
Feature Cues \(E_g + E_f\) (w/o \(E_r\)) -1.8 -0.1 +0.1 Retains most object-level performance

Key Findings

  • Reprojected Appearance Features Serve as the Primary Driver: As revealed by Table 6, reprojected appearance features (\(E_{\text{feat}}\)) are the single most indispensable cue. Isolating geometric depth differences (\(E_{\text{geom}}\)) preserves pixel IoU (+2.1) but causes obj/im AP to plunge by 12.1 points (from 43.8% to 31.7%) and obj/sc AP to fall by 10.3 points, as pure depth cannot discriminate flat object displacements and is susceptible to surface noise.
  • Training-Free Foundation Model Ensemble Surpasses Task-Specific Supervised Models: Operating entirely zero-shot, SceneDiff achieves 43.8% obj/im AP on SD-V, outperforming CYWS-2D fine-tuned on the validation split (24.5%). On the two-view RC-3D benchmark, SceneDiff achieves 65.3% AP50, surpassing CYWS-3D using ground-truth sensor depth (50.0%) by 15.3 points.
  • Inference Efficiency Significantly Outperforms Render-and-Compare Approaches: Table 3 records an average runtime of 159.4 seconds on a single A40 GPU for a 10-second sequence pair (reducible to 65 seconds under bf16 batching and 19.2GB VRAM), whereas 3DGS-based pipelines (3DGS-CD and MV3DCD) require over 30 minutes of per-scene optimization and frequently collapse under wide camera trajectory deviations.

Highlights & Insights

  • Asymmetric Visibility Geometry Gating: Exploiting the directional ray nature of scene capture, treating only positive depth reprojection differences as valid cues while masking negative discrepancies effectively eliminates the geometric ambiguity between real scene changes and standard foreground occlusions.
  • Modular Foundation-Model Cascade Without Task-Specific Training: Seamlessly chaining feed-forward geometry estimation (\(\pi^3\)), class-agnostic segmentation (SAM), and dense self-supervised representations (DINOv3) eliminates the risk of overfitting and naturally gains performance as underlying visual foundation models advance.
  • Closed-Loop Physical Robotic Validation: The authors demonstrate practical viability by deploying SceneDiff on a physical manipulator arm to clear a cluttered workspace back to its pre-recorded baseline state, picking up newly introduced trash and restoring displaced objects.

Limitations & Future Work

  • Inherent Ambiguity in Symmetrical Object Permutations: When visually identical objects are swapped (e.g., swapping two identical soda cans), geometry and semantic representations cannot disambiguate the permutation from a single object displacement, leading to conservative minimal-motion predictions.
  • Sensitivity to Extreme Lighting and Textureless Environments: As the entire framework builds upon the pose and depth estimation of \(\pi^3\), failure modes in extreme low-light, blank textureless surfaces, or paths with negligible co-visibility directly compromise the downstream change scoring pipeline.
  • Absence of Semantic State Change and Micro-Deformation Tracking: The algorithm assumes rigid object-level additions, removals, or translations; detecting subtle semantic state evolutions (e.g., raw bread toasted into toast, liquid levels in cups) and surface-level damage remains an open problem.
  • vs CYWS-2D / CYWS-3D: Prior learning-based methods rely on synthetic training pairs and struggle under real camera trajectory drifts; SceneDiff anchors both sequences into a metric 3D space via feed-forward joint geometry, achieving generalizable change detection without retraining.
  • vs 3DGS-CD / MV3DCD: 3DGS and NeRF change detection frameworks require dense multi-view optimization taking over 30 minutes per sequence and fail when novel view synthesis extrapolates poorly; SceneDiff operates via single-pass feed-forward inference and region-level feature comparison, reducing runtime by more than 10x.

Rating

  • Novelty: โญโญโญโญโ˜† Introduces the first multi-trajectory video change detection benchmark with instance annotations, paired with an elegant asymmetric training-free alignment framework.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive benchmarking across diverse indoor/outdoor scenes, egocentric kitchen sequences, two-view datasets, detailed ablations, and real-world robotic demonstration.
  • Writing Quality: โญโญโญโญโญ Well-structured, mathematically rigorous, with clean diagrams and reproducible specifications.
  • Value: โญโญโญโญโญ Provides an essential dataset, annotation interface, and strong baseline for 3D spatial intelligence, digital twins, and embodied robotics.