Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/tj12323/SGC
Area: Video Generation
Keywords: Video Generation, 3D Geometric Consistency, Dynamic Video Evaluation, Camera Pose Estimation, Background Rigidity

TL;DR¶

Addressing the insensitivity of existing video generation metrics to geometric distortions and their over-penalization of valid dynamic motion, this paper introduces SGC, a diagnostic metric that decouples static background from dynamic foreground and measures multi-region camera pose divergence to accurately quantify 3D spatial geometric consistency.

Background & Motivation¶

Recent generative video synthesis models have achieved remarkable visual fidelity, producing dynamic sequences that closely approach photorealism. However, even when generated videos score exceptionally well on fidelity-centric metrics like FVD (Fréchet Video Distance) and per-frame aesthetic quality, they frequently exhibit severe 3D spatial geometric inconsistencies across frames. These foundational failures manifest in several ways: geometric warping where rigid structures unnaturally distort during camera translation; incoherent motion where static scene elements illogically fuse with moving foreground objects; object impermanence where persistent landmarks flicker or morph; and perspective failures where distant backgrounds fail to follow 3D projective geometry.

The primary barrier to detecting and diagnosing these spatial inconsistencies lies in fundamental design biases of existing evaluation frameworks. Fidelity-centric metrics such as FVD and CLIPScore suffer from a content-over-motion bias; dominated by per-frame visual appearance, they remain largely insensitive to structural deformities and reward visually pleasing textures despite underlying spatial collapse. Conversely, consistency-focused metrics imported from novel view synthesis (e.g., TSED, MEt3R) or general benchmarks (e.g., VBench Background Consistency) suffer from a fragility-to-motion bias. They either fail when complex dynamic objects enter the scene or conflate valid foreground dynamics with background instability. Consequently, prior benchmarks cannot reliably isolate and evaluate the 3D stability of the static environment amidst complex dynamic motion.

This paper is grounded in a fundamental physical principle: in any physically coherent 3D scene, the apparent motion of all static background elements must adhere to a single, shared camera transformation. If a generative model preserves authentic 3D geometry, relative camera poses independently estimated from distinct background sub-regions must strictly agree; any divergence directly signals geometric inconsistency. Core idea: isolate the static background via long-range point-track motion segmentation, partition it into depth-coherent sub-regions, estimate local camera poses via PnP, and quantify 3D spatial geometric consistency through an objective PCA-weighted aggregation of inter-region pose variance, global trajectory alignment, and cross-frame depth warping error (SGC).

Method¶

Overall Architecture¶

SGC (Spatial Geometric Consistency) is a feedforward diagnostic metric designed to evaluate 3D geometric consistency between consecutive frames $(f_{i-1}, f_i)$ in dynamically generated videos. The pipeline operates without manual intervention, disentangling moving objects from the static scene, inferring dense geometry, computing multi-scale relative poses, and aggregating spatial variances into a unified score.

The end-to-end evaluation flows through four principal stages: 1. Static Background Isolation: Input frames are processed by a dynamic motion segmentation network to extract foreground moving masks, whose complement yields the static environment mask; 2. Global Geometry and Depth Estimation: A geometry foundation model infers global camera poses and intrinsics, while a video depth model extracts temporally consistent dense depth maps; 3. Depth Stratification & Local Pose Estimation: The static background is partitioned into distinct depth clusters, and local relative camera poses are solved for each sub-region using dense 2D tracks and backprojected 3D points via PnP; 4. Multi-Scale Variance Aggregation: Inter-segment local pose variance, global pose alignment discrepancy, and cross-frame depth warping error are computed and objectively aggregated via PCA first-component loadings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RGB Video Frames"] --> B["Static Background Isolation<br/>SegAnyMo extracts dynamic masks to isolate static scene"]
    A --> C["Geometry & Depth Estimation<br/>VGGT global poses + Video Depth Anything depth maps"]
    B --> D["Depth Stratification & Local Pose Estimation<br/>1D depth clustering + 2D tracking & PnP pose solve"]
    C --> D
    D --> E["Multi-Scale Variance & Depth Error<br/>Local inter-segment variance + global alignment + depth warp"]
    C --> E
    E --> F["PCA Objective Weighting<br/>Outputs scalar SGC metric score"]

Key Designs¶

1. Static Background Isolation: Disentangling dynamic foreground to resolve fragility-to-motion bias

A pervasive flaw in existing consistency metrics is their vulnerability to foreground object motion: evaluating unsegmented frames inadvertently penalizes valid, complex object kinematics as background distortion, while severe object deformation simultaneously obscures an otherwise stable background geometry. SGC overcomes this by introducing SegAnyMo as a preprocessing stage. By classifying long-range spatio-temporal point tracks, SegAnyMo generates per-frame moving-object masks $\mathcal{M}_{dyn}$. The static background region $\mathcal{M}_{static}$, which serves as the exclusive domain for geometric inspection, is defined as the spatial complement: $$\mathcal{M}_{static} = \Omega \setminus \mathcal{M}_{dyn}$$ where $\Omega$ denotes the full image spatial domain. Isolating $\mathcal{M}_{static}$ ensures that all subsequent depth clustering, feature tracking, and camera pose estimations evaluate strictly rigid environmental structures.

2. Depth-Aware Stratification & Local PnP Pose Estimation: Formulating rigidity verification as local pose consensus

In a physical scene, background elements span diverse depths (e.g., foreground road, mid-ground architecture, distant terrain), yet their 2D optical flows are governed by an identical camera motion. Traditional Structure-from-Motion (e.g., COLMAP) frequently collapses on generative videos due to subtle texture drift. SGC leverages Video Depth Anything to predict temporally smooth dense depth $D_k$, alongside VGGT for global intrinsics $K_k$ and camera poses. Across $\mathcal{M}_{static}$ in frame $f_i$, valid pixel depths are grouped via GPU-accelerated 1D k-means clustering into $N_s$ depth strata $s_j$.

Within each sub-region $s_j$, DELTA tracks dense 2D feature correspondences $\{p_{i-1}^{(m)} \leftrightarrow p_i^{(m)}\}$ between frames $f_{i-1}$ and $f_i$. Unprojecting $p_{i-1}^{(m)}$ using depth $D_{i-1}$ and intrinsic matrix $K_{i-1}$ yields 3D camera-frame coordinates: $$P_{i-1}^{(m)} = D_{i-1}(p_{i-1}^{(m)}) K_{i-1}^{-1} [p_{i-1}^{(m)\top}, 1]^\top$$ The Perspective-n-Point (PnP) algorithm with RANSAC then independently solves for the local relative rotation $R_i^{loc, j} \in \mathrm{SO}(3)$ and translation $t_i^{loc, j} \in \mathbb{R}^3$ for each sub-region. Notably, 1D depth clustering permits non-contiguous surfaces sharing the same depth plane to group together, naturally broadening the spatial baseline of correspondences across the image plane and substantially improving PnP conditioning.

3. Multi-Scale Variance and Depth Consistency Metrics: Quantifying local divergence and global drift

If the video preserves authentic 3D rigidity, camera transformations independently derived from disparate static regions should perfectly coincide and agree with the scene's global trajectory. SGC captures geometric collapse through three complementary error terms: - Local Inter-Segment Consistency: Measures the divergence among local relative poses $\{P_i^{loc, j}\}$. Rotational dispersion is quantified using geodesic distance variance: $$\sigma_{rot, loc}^2 = \frac{1}{N_s} \sum_{j=1}^{N_s} \left( d_\theta(R_i^{loc, j}, \bar{R}_i^{loc}) \right)^2, \quad d_\theta(R_A, R_B) = \arccos\left(\frac{\mathrm{Tr}(R_A^\top R_B) - 1}{2}\right)$$ while translational dispersion is evaluated as Euclidean squared variance $\sigma_{trans, loc}^2$ relative to the mean translation $\bar{t}_i^{loc}$. - Global Pose Consistency: Evaluates the alignment between each local sub-region motion and the global camera transformation $P_i^{glo} = (R_i^{glo}, t_i^{glo})$ derived from VGGT, computing global rotational variance $\sigma_{rot, glob}^2$ and global translational variance $\sigma_{trans, glob}^2$. - Depth Consistency Error: Aligns frame $f_{i-1}$ to $f_i$ using global pose $P_i^{glo}$ and reprojects depth to compute absolute discrepancy over the overlapping valid static mask $\mathcal{V}$: $$E_{depth} = \frac{1}{|\mathcal{V}|} \sum_{u \in \mathcal{V}} |D_{i-1 \to i}(u) - D_i(u)|$$

4. PCA-Based Objective Weighting: Data-driven metric aggregation without manual tuning

To aggregate the five atomic component metrics without subjective bias, SGC standardizes raw values via Z-score and min-max scaling to a unified $[0, 1]$ range (where 0 indicates optimal consistency). A Principal Component Analysis (PCA) is performed on the normalized score matrix across a diverse calibration dataset. The normalized loadings of the first principal component (PC1) are assigned as fixed objective weights $w_k$, computing the final scalar score as: $$\mathrm{SGC}_d = \sum_{k=1}^{N_M} w_k M''_{d, k}$$ This procedure automatically places greater emphasis on sub-metrics capturing the dominant shared variance of geometric breakdown, ensuring strong transferability and diagnostic robustness across video domains.

Key Experimental Results¶

Main Results¶

The evaluation benchmark encompasses 1,296 videos, including 300 real-world reference videos across three distinct domains (nuScenes autonomous driving, RT-1 robotic manipulation, OpenVid diverse web captures) and 996 synthetic videos generated by 10 leading models on GenWorld. SGC is compared against MEt3R, MEt3R with moving object segmentation (MEt3R(+MOS)), FVD, VBench dimensions (Background Consistency BC, Dynamic Degree DD, weighted composite), TRAJAN, and FVMD.

Method	Task	SGC ↓	MEt3R ↓	MEt3R(+MOS) ↓	FVD ↓	VBench-BC ↑	VBench-DD ↑	VBench-w ↑	TRAJAN ↑	FVMD ↓
Cosmos	V2V	0.0722	0.0741	0.2350	760.32	0.9461	0.6462	0.6037	0.6023	15389.47
OpenSora-t	T2V	0.0831	0.0919	0.2513	838.04	0.9259	0.8286	0.5370	0.2835	13169.15
Zeroscope	T2V	0.0912	0.1558	0.4602	823.14	0.9326	0.5287	0.4974	0.5230	18714.92
VideoCrafter	T2V	0.0973	0.1176	0.2205	699.71	0.9650	0.6320	0.7114	0.6344	16125.86
Hotshot	T2V	0.1172	0.1371	0.3568	908.05	0.9468	0.8190	0.6801	0.5518	-
Lavie	T2V	0.1241	0.1122	0.2793	679.82	0.9576	0.7559	0.7090	0.6504	14424.44
OpenSora-i	I2V	0.1631	0.1030	0.3088	751.21	0.8993	0.8333	0.3638	0.3208	9250.18
Seine	I2V	0.2837	0.2613	0.6343	808.96	0.8891	0.8981	0.1215	0.4669	23555.54
Modelscope	T2V	0.3129	0.1851	0.4208	784.67	0.9196	0.7966	0.3313	0.5962	19511.57
Latte	T2V	0.3226	0.1707	0.2601	801.01	0.9412	0.8559	0.4403	0.4418	19561.51
OpenVid (Real)	-	0.0530	0.0371	0.0486	752.26	0.9408	0.6300	0.8330	0.4869	-
nuScenes (Real)	-	0.0613	0.0485	0.0753	1297.07	0.9080	0.8800	0.5370	0.2320	-
RT-1 (Real)	-	0.0639	0.0799	0.0698	1352.88	0.9229	0.9800	0.6499	0.7362	-

Ablation Study¶

The ablation study validates SGC architectural choices by comparing variations across all 10 generative models and 3 real-world datasets: removing moving object segmentation (w/o MOS), replacing depth clustering with a fixed spatial grid (Grid depth), enforcing depth plus spatial contiguity (Depth + Spatial), and the full pipeline (SGC-full).

Variant	Cosmos	Hotshot	Latte	Lavie	Modelscope	OpenSora-i	OpenSora-t	Seine	VideoCrafter	Zeroscope	OpenVid	RT-1	nuScenes	Note
w/o MOS	0.052	0.104	0.286	0.133	0.222	0.157	0.080	0.259	0.086	0.075	0.014	0.163	0.064	Severe degradation on RT-1 (+155% error) due to unsegmented robot motion
Grid depth	0.067	0.109	0.353	0.170	0.314	0.190	0.107	0.306	0.101	0.071	0.062	0.093	0.163	Naive spatial grids fragment co-planar surfaces, inflating nuScenes error to 0.163
Depth + Spatial	0.082	0.133	0.555	0.161	0.483	0.205	0.101	0.365	0.086	0.084	0.019	0.051	0.153	Enforcing contiguity restricts spatial point spread, destabilizing local PnP
SGC-full	0.072	0.117	0.323	0.124	0.313	0.163	0.083	0.284	0.097	0.091	0.053	0.064	0.061	Optimal balance across dynamic scenes, planar structures, and real anchors

Key Findings¶

2D Fidelity Blind to 3D Collapse: Models such as Latte and Modelscope achieve competitive FVD scores (801.0 and 784.7) and high VBench Background Consistency ratings (0.941 and 0.920), yet exhibit severe 3D geometric breakdown with SGC scores of 0.3226 and 0.3129. This confirms that 2D textural plausibility frequently conceals multi-plane geometric collapse.
MOS is Essential in Dynamic Settings: On the robotic interaction dataset RT-1, removing MOS inflates error from 0.064 to 0.163 (+155%), proving that failing to isolate moving objects erroneously penalizes valid physical action as geometric instability.
1D Depth Clustering Preserves PnP Conditioning: Constraining sub-regions to be spatially contiguous (Depth + Spatial) or adopting fixed grids (Grid depth) fragments co-planar structures like roads into narrow regions, degrading PnP numerical stability. 1D depth clustering allows non-contiguous co-planar patches to combine, maximizing spatial baseline span.
Robustness to Camera Motion Dynamics: In real-world videos, the correlation between Estimated Camera Dynamics (ECD) and SGC is weak to moderate ($R^2 \le 27\%$), and real videos consistently maintain low SGC scores ($<0.5$) even under extreme camera velocities. This validates that SGC penalizes structural deformation rather than camera speed.

Highlights & Insights¶

First-Principles Rigidity Constraint: Instead of treating consistency as generic feature-matching distance, SGC formulates evaluation on rigid Euclidean transformations: all static 3D points must obey a single camera pose change.
Multi-Region Local PnP Pose Divergence: Formulating geometric distortion as Lie group rotation and translation variance across depth-stratified sub-regions provides explicit physical meaning and actionable diagnostic signals.
High Estimator Invariance: Replacing the underlying pose estimator VGGT with dynamic reconstruction models (Any4D, Page4D) or omitting global pose preserves strong rank correlation (Spearman $\rho \ge 0.860$), demonstrating that SGC assesses true generator weaknesses rather than estimator artifacts.

Limitations & Future Work¶

Dependency on Upstream Foundation Models: SGC chains motion segmentation (SegAnyMo), dense 2D tracking (DELTA), and monocular depth estimation (Video Depth Anything). In scenes with extreme underexposure, intense specular reflections, or severe motion blur, upstream perception failures may degrade PnP precision.
Strict Rigidity Assumption: The formulation assumes the background is predominantly rigid. For environments featuring massive non-rigid backgrounds (e.g., turbulent open ocean, wildfire, dense foliage swaying under violent storms), natural environmental motion might be misclassified as geometric failure.
Future Directions: Exploring end-to-end self-supervised geometric consistency estimators that distill discrete multi-stage pipelines into a unified spatio-temporal geometric representation.

vs FVD / CLIPScore: Conventional metrics measure statistical distributions of deep neural features, suffering from content-over-motion bias; SGC explicitly reconstructs 3D projective transformations to diagnose true structural stability.
vs VBench (Background Consistency): VBench-BC relies on frame-wide 2D feature similarities, which reward over-smoothed and static textures; SGC enforces rigid 3D pose agreement and depth consistency across multiple depth layers.
vs MEt3R / TSED (Novel View Synthesis Metrics): Prior consistency metrics assume entirely static scenes and break down under independent foreground motion; SGC decouples dynamic objects via trajectory tracking, serving as the first general geometric metric for dynamic video synthesis.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering formulation using multi-region local PnP pose divergence to measure 3D geometric consistency in dynamic video generation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking over 10 generative models, 3 real-world datasets (1,296 videos), controlled synthetic perturbations, and estimator ablations]
Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear physical intuition, cohesive prose, and well-structured empirical validation]
Value: ⭐⭐⭐⭐⭐ [Fills a critical void in generative video evaluation, providing an indispensable diagnostic tool for building 3D-consistent world models]