SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://smg-gaussian.github.io/
Area: 3D Vision
Keywords: Dynamic Gaussian Splatting, Monocular Dynamic Reconstruction, Semantic Motion Graph, Confidence-Aware ARAP, Local Rigid Motion Control

TL;DR¶

SMG models monocular dynamic Gaussian deformation via a Semantic Motion Graph that restricts motion propagation within semantic groups and incorporates confidence-aware ARAP along with local rigid motion control to guide weakly constrained regions, achieving state-of-the-art novel-view synthesis and robust 3D tracking.

Background & Motivation¶

Reconstructing dynamic 3D scenes and synthesizing high-fidelity novel views from casual monocular videos is a cornerstone capability for spatial computing, AR/VR, and robotics. Following the breakthrough of 3D Gaussian Splatting (3DGS) in static scene representation, multiple extensions have adapted explicit Gaussian primitives to model time-varying radiance fields. However, monocular capture introduces severe fundamental ill-posedness: camera motion and non-rigid object dynamics are deeply intertwined, and pervasive occlusions leave substantial portions of the 3D volume unobserved from the single input viewpoint. Under sparse supervision, Gaussian primitives readily collude—adjusting their opacities, scales, and colors to overfit training camera rays while hallucinating distorted geometry and floating artifacts in novel viewpoints.

To regularize these under-constrained deformations, recent methods construct 3D lifting fields using off-the-shelf single-view priors (e.g., metric depth and 2D point trajectories) and enforce local geometric smoothness or physical deformation constraints. Nonetheless, relying solely on Euclidean spatial proximity reveals critical limitations in real-world scenarios. In practice, spatial closeness does not imply shared dynamics: when an actor's hand moves closely past a stationary tabletop or a stationary mouse, spatial nearest-neighbor connections mistakenly propagate fast hand motion into static objects, causing severe boundary blurring and motion leakage. Furthermore, upstream monocular depth and tracking priors inherently suffer from noise and tracking failures; unweighted regularizations indiscriminately broadcast such noisy estimates throughout the deformation field.

This work builds upon the foundational observation that physical motion coherence in natural environments is fundamentally structured by semantic coherence. Gaussian primitives belonging to the same semantic component naturally share coordinated physical dynamics. The core idea is to anchor dynamic Gaussian deformation to a Semantic Motion Graph (SMG) governed by trajectory-level semantic gating, and deploy confidence-aware ARAP alongside local rigid motion control so that reliable nodes guide under-constrained regions while preserving topological and physical fidelity.

Method¶

Overall Architecture¶

SMG decomposes monocular dynamic scene reconstruction into a structured pipeline: first, 2D tracking trajectories and metric depth predictions from off-the-shelf models are lifted into 3D spacetime, where edges are established by combining local spatial proximity with DINOv3 semantic gating; second, motion uncertainty is mitigated via confidence-aware as-rigid-as-possible (C-ARAP) modeling and local rigid motion (LRM) control, ensuring that reliable nodes guide less-constrained regions; finally, canonical 3D Gaussians are anchored to local SMG nodes and deformed via dual-quaternion blending, trained end-to-end under joint photometric and geometric supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video Input<br/>RGB frames + Metric Depth + 2D Long-range Tracks"] --> B["Semantic Motion Graph Construction<br/>DINOv3 Semantic Gating + Spatiotemporal KNN Topology"]
    B --> C["Confidence-Aware ARAP (C-ARAP)<br/>Visibility & Rigidity-based Asymmetric Edge Modulation"]
    C --> D["Local Rigid Motion Control (LRM)<br/>Neighborhood Twist Estimation & Robust Huber Penalty"]
    D --> E["SMG-Driven Gaussian Deformation<br/>Dual-Quaternion Blending (DQB) over Local Nodes"]
    E --> F["Joint Photometric & Geometric Optimization<br/>RGB, Depth, Dynamic Mask & 3D Track Loss"]

Key Designs¶

1. Semantic Motion Graph Construction: Restricting Motion Propagation to Coherent Semantic Groups To prevent motion leakage across distinct entities that happen to be physically adjacent, SMG constructs graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ over lifted 3D trajectories $\mathcal{V}$. Edges between nodes $i$ and $j$ are established only when both geometric proximity and semantic compatibility criteria are satisfied: $$\mathcal{E}_{ij} = \mathbf{1}\left[j \in \mathrm{KNN}(i)\right] \cdot \mathbf{1}\left[\cos(f_i, f_j) \ge \tau\right]$$ Here, $\mathrm{KNN}$ is computed at the trajectory level using Euclidean distances across co-visible frames, and $\tau$ denotes the semantic gating threshold. The semantic descriptor $f_i$ is extracted by projecting trajectory positions onto 2D image coordinates, sampling feature maps from DINOv3, and aggregating them via top-k cosine-consistent averaging. This mechanism cuts off spurious connections between interacting objects (e.g., a moving hand and a static laptop), constraining motion regularization strictly within the appropriate physical boundary.

2. Confidence-Aware ARAP: Guiding Unconstrained Regions via Reliable Anchors Off-the-shelf monocular trackers and depth estimators inevitably produce noisy predictions, especially under severe occlusions. Imposing standard symmetric As-Rigid-As-Possible (ARAP) regularization causes erroneous motions to corrupt stable regions. SMG introduces Confidence-Aware ARAP (C-ARAP), where node-level confidence $c_{t,i}$ is computed per frame from visibility $\mathrm{viz}_{t,i}$, velocity deviation from the local neighborhood $r_{t,i}^v$, and anomalous edge length variation $r_{t,i}^\ell$: $$c_{t,i} = \mathrm{viz}_{t,i} \cdot \left(c_{t,i}^{\text{motion}}\right)^{\lambda_m} \cdot \left(c_{t,i}^{\text{rigid}}\right)^{\lambda_r}$$ This confidence is mapped to asymmetric directed edge weights $\tilde{w}_{ij}^t = w_{ij}^{\text{topo}} \cdot (\alpha + (1-\alpha) c_{t,j})$. The resulting C-ARAP objective ensures that highly reliable nodes actively steer uncertain neighbors to maintain local orientation and edge lengths, while unreliable or occluded nodes are prevented from polluting well-observed structures.

3. Local Rigid Motion Control: Constraining Local Velocity Fields with Physical Twists In weakly supervised regions, unconstrained gradient steps can induce irregular velocity fluctuations that tear the Gaussian cloud. SMG introduces Local Rigid Motion Control (LRM) to enforce that local velocity fields adhere to a rigid twist. By calculating the weighted center of mass and average linear velocity within node $i$'s neighborhood, the local angular velocity $\omega_i^t$ is estimated using weighted least squares: $$\omega_i^t = \arg\min_\omega \sum_{j \in \mathcal{N}(i)} \tilde{w}_{ij}^t \left\| \tilde{v}_{ij}^t - \omega \times \tilde{x}_{ij}^t \right\|^2$$ Residual velocities that deviate from this optimal rigid twist are penalized using a robust Huber loss $\rho(\cdot)$ in $\mathcal{L}_{\text{LRM}}$. This regularizer damps unphysical high-frequency motion artifacts while accommodating legitimate articulated deformation.

4. SMG-Driven Gaussian Deformation and Joint Optimization: Smooth Articulation with Multi-Prior Supervision During scene initialization, dynamic 3D Gaussians are back-projected from valid foreground depth pixels at reference time $t_i^{\text{ref}}$ and anchored to their nearest SMG node. To prevent harsh surface creases at cluster boundaries, each Gaussian is steered by $K$ neighboring graph nodes using Dual-Quaternion Blending (DQB), yielding smooth, continuous spatial transformations over time. The complete model is optimized end-to-end using photometric rendering loss $\mathcal{L}_{\text{rgb}}$, metric depth loss $\mathcal{L}_{\text{dep}}$, dynamic mask loss $\mathcal{L}_{\text{mask}}$, and 3D track alignment loss $\mathcal{L}_{\text{track}}$, coupled with density control for adaptive primitive allocation.

Loss & Training¶

The overall training objective combines photometric and geometric terms: $$\mathcal{L} = \lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}} + \lambda_{\text{dep}}\mathcal{L}_{\text{dep}} + \lambda_{\text{mask}}\mathcal{L}_{\text{mask}} + \lambda_{\text{track}}\mathcal{L}_{\text{track}} + \lambda_{\text{C-ARAP}}\mathcal{L}_{\text{C-ARAP}} + \lambda_{\text{LRM}}\mathcal{L}_{\text{LRM}}$$ Optimization parameters are set to $K=16$ nearest neighbors, confidence lower bound $\alpha=0.6$, soft tolerance thresholds $\kappa_m = \kappa_r = 2.5$, and semantic gating threshold $\tau=0.75$ for Dycheck and NVIDIA datasets ($\tau=0.85$ for the SMG dataset). The system is trained per scene on an NVIDIA H100 GPU using PyTorch.

Key Experimental Results¶

Main Results¶

The authors evaluate SMG on the Dycheck (iPhone) benchmark across all 7 scenes at half resolution, comparing against leading dynamic NeRF and 3DGS baselines. SMG achieves top-tier rendering accuracy across all metrics and excels at 3D point trajectory tracking.

Table 1: Novel view synthesis comparison on the Dycheck (iPhone) dataset

Method	mPSNR ↑	mSSIM ↑	mLPIPS ↓
T-NeRF	16.96	0.577	0.379
NSFF	15.46	0.551	0.396
4DGS	13.64	-	0.428
DyBluRF	17.37	0.591	0.373
Shape-of-Motion	17.32	0.598	0.296
MoSca	19.32	0.706	0.264
OriGS (reproduced)	19.43	0.695	0.281
OriGS* (reported)	19.69	0.716	0.256
SMG (Ours)	19.54	0.718	0.250

Table 3: 3D tracking performance on Dycheck dataset (5 scenes)

Method	EPE ↓	$\delta_{3D}^{.05}$ (%) ↑	$\delta_{3D}^{.10}$ (%) ↑
MoSca	0.055	73.1	89.6
OriGS	0.057	71.9	89.7
SMG (Ours)	0.052	74.1	91.6

Ablation Study¶

Ablation experiments conducted on four challenging Dycheck scenes (Apple, Spin, Space-out, Wheel) isolate the contribution of each core component.

Table 4: Ablation study of individual pipeline components

Config	PSNR ↑	SSIM ↑	LPIPS ↓	Note
Base (4DGS)	12.92	0.451	0.598	Collapses under novel viewpoints without motion regularization
SMG-only	19.07	0.700	0.280	Semantic graph topology alone provides substantial +6.15 dB gain
w/o C-ARAP	19.09	0.703	0.281	Removing confidence weighting leads to topological drift
w/o LRM	19.96	0.731	0.248	Removing rigid twist control introduces local velocity jitter
Full SMG	20.11	0.738	0.242	Full integration achieves best perceptual and structural scores

Key Findings¶

Semantic gating eliminates cross-object motion artifacts: In challenging sequences like "Laptop2", baselines propagate user arm motions directly to the adjacent static mouse, causing it to float away in novel views. SMG strictly preserves object boundaries by rejecting cross-semantic graph edges via DINOv3 feature similarity.
Synergy between C-ARAP and LRM: Removing C-ARAP degrades PSNR back to 19.09 dB, proving that asymmetric confidence weighting is vital when dealing with noisy priors. Meanwhile, LRM suppresses residual high-frequency velocity jitter, further lowering LPIPS from 0.248 to 0.242.
Robustness under wide-baseline ego-exo setups: On the proposed SMG Dataset featuring rapid egocentric camera motion paired with static exocentric views, SMG consistently outperforms prior state-of-the-art methods across all scenes (overall PSNR 13.74 vs MoSca 13.31, SSIM 0.559 vs 0.542).

Highlights & Insights¶

Transitioning from geometric proximity to semantic manifolds: Conventional dynamic reconstruction architectures assume that spatially close points should deform similarly. SMG illustrates that semantic coherence is the true physical driver of local rigidity, opening new avenues for integrating foundation model representations into low-level geometry pipelines.
Asymmetric confidence flow resolves prior contamination: Rather than discarding imperfect monocular priors, C-ARAP converts topological regularization into an asymmetric information flow: high-confidence tracked regions anchor the scene, while occluded regions are passively guided without polluting the global graph.
Challenging ego-exo benchmark for dynamic scene modeling: Introducing the SMG Dataset fills an important gap in dynamic evaluation, testing dynamic 3DGS under authentic, large-baseline human-object interactions where standard front-facing assumptions completely break down.

Limitations & Future Work¶

Dependency on upstream vision foundation models: Performance remains bounded by the discriminative quality of DINOv3 features and metric depth estimators (such as Depth Anything 3). Systemic scale drift or loss of fine-grained semantic distinctions in complex clutter can degrade edge connectivity.
Per-scene optimization runtime: Training individual per-scene dynamic Gaussian fields requires substantial GPU compute, preventing deployment in real-time interactive systems. Future explorations could combine feed-forward 4D Gaussian transformers with semantic graph structures to achieve generalized zero-shot dynamic reconstruction.

vs MoSca [CVPR 2024] / Shape-of-Motion [ECCV 2024]: While these methods construct motion scaffolds from lifted 2D trajectories, their graph topology relies entirely on spatial distance and uniformly weights all nodes. SMG introduces semantic feature gating and confidence-aware asymmetric regularization, fundamentally preventing motion contamination.
vs OriGS [2025]: OriGS focuses on orientation consistency but remains vulnerable to noisy priors under sparse views. SMG demonstrates that explicit physical velocity constraints (LRM) combined with semantic graphs offer superior stability against geometric collapse.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Pioneering integration of trajectory-level vision foundation semantics into dynamic Gaussian graphs with asymmetric confidence modulation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across Dycheck, NVIDIA, and a dedicated ego-exo dataset, encompassing novel view synthesis, 3D tracking, and modular ablations]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural flow with crisp technical exposition, rigorous formulation, and compelling qualitative evidence]
Value: ⭐⭐⭐⭐☆ [Offers a highly practical framework and benchmark for robust dynamic scene capture in robotics and egocentric vision]