Relaxed Rigidity with Ray-based Grouping for Dynamic Gaussian Splatting

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Dynamic 3D Gaussian Splatting, Ray-based Grouping, Motion Regularization, Spectral Regularization, Relaxed Rigidity

TL;DR

Addressing error accumulation from external motion priors and artificial over-constraint in traditional KNN-based rigid regularization, this work clusters visible Gaussians using ray blending weights and applies motion coherence and covariance eigenvalue spectral regularization, achieving consistent state-of-the-art dynamic reconstruction with zero inference overhead.

Background & Motivation

Reconstructing dynamic 3D scenes from monocular video sequences using 3D Gaussian Splatting (3DGS) has shown great potential for photo-realistic novel view synthesis. However, monocular inputs intrinsically lack cross-view geometric cues and explicit multi-view dynamic constraints. As a result, optimization of dynamic Gaussian primitives is inherently under- or over-constrained, frequently leading to physically implausible displacements, chaotic Gaussian trajectories, and degraded temporal consistency characterized by severe floaters and geometric distortion.

To alleviate this ambiguity, prevailing dynamic 3DGS frameworks predominantly follow two paradigms: relying on external 2D/3D motion priors (such as optical flow or 2D point tracks) or imposing strict local rigidity constraints on K-Nearest Neighbors (KNN) groups (e.g., via As-Rigid-As-Possible / ARAP formulations). Nevertheless, both strategies exhibit fundamental shortcomings. External proxy priors are formulated in 2D image coordinates rather than true 3D geometry; in the presence of specular reflections, textureless regions, or rapid view-dependent occlusions, external tracking and flow estimates fail and propagate erroneous gradients into 3D scene optimization. Meanwhile, Euclidean KNN clustering ignores Gaussian primitive properties such as scale, opacity, and transmission, grouping structurally distinct or occluded regions together. Furthermore, strict local rigidity assumptions penalize natural non-rigid deformations and topological transitions, crushing fine details.

The key insight of this paper is to leverage the rasterization process itself as an implicit, occlusion-aware visibility group filter, and to relax strict point-to-point rigidity into directional alignment coupled with distribution-level shape preservation. Core idea: cluster spatially contiguous, unoccluded Gaussians along viewing rays via their alpha-blending weights, and enforce relaxed rigidity by combining magnitude-free motion directional consistency with eigenvalue spectral regularization on local position covariances, thereby establishing physically consistent dynamic representations without external priors.

Method

Overall Architecture

The proposed approach is a model-agnostic, prior-free regularization framework that can be seamlessly incorporated into representative dynamic 3DGS architectures, including deformation-field formulations (e.g., RTD, MoDec-GS, Grid4D) and basis-trajectory models (e.g., Ex4DGS). During forward volume rendering, viewing rays gather visible Gaussians whose contribution weights surpass a visibility threshold. Two complementary relaxed rigidity constraints are then computed over these groups across time intervals: motion coherence regularization to align movement directions, and spectral regularization to maintain 3D spatial shape statistics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video & Dynamic Gaussians<br/>Time-varying position and attributes"] --> B["Ray-based Gaussian Grouping<br/>Sorted rays & alpha-blending filter"]
    B --> C["Motion Coherence Regularization<br/>Group-mean cosine directional alignment"]
    B --> D["Spectral Regularization<br/>Welford single-pass covariance & eigenvalue matching"]
    C --> E["Joint Optimization & Relaxed Rigidity<br/>Photometric loss with geometric regularizers"]
    D --> E
    E --> F["Temporally Coherent 4D Dynamic Scene<br/>Zero-overhead real-time rendering"]

As illustrated in the overall pipeline, the rasterizer sorts Gaussians along camera rays. The attenuation of accumulated transmittance naturally eliminates occluded primitives, isolating coherent surface clusters. The two-pronged regularization subsequently aligns their trajectory directions and preserves local spatial covariance eigenvalues across time, preventing non-physical scattering while accommodating flexible non-rigid deformations.

Key Designs

1. Ray-based Gaussian Grouping: Repurposing rasterization visibility for occlusion-aware clustering

Rather than executing computationally expensive spatial queries in 3D Euclidean space with KNN—which blindly groups points across occluding depth discontinuities—the method exploits the native alpha-blending mechanism in 3DGS. For each pixel \(p_j\), sorted Gaussians along the ray contribute with weight \(w_i = T_i (1 - e^{-\alpha_i})\), where \(T_i = \exp(-\sum_{j=1}^{i-1} \alpha_j)\). The ray-based group \(\mathcal{N}_j\) is defined by:

\[\mathcal{N}_j = \{ \mathcal{G}_i \mid w_i > \tau \}\]

Because transmittance \(T_i\) drops sharply behind opaque front surfaces, thresholding by \(\tau\) filters out background primitives and isolates spatially contiguous Gaussians on the visible surface. This grouping is obtained as a zero-cost byproduct of standard volume rendering, dynamically adapting cluster sizes to local scene complexity without manual scale heuristics.

2. Motion Coherence Regularization (MCR): Relaxing displacement magnitudes for directional trajectory alignment

Within each ray group \(\mathcal{N}_j\), imposing uniform rigid displacement vectors \(\mathbf{d}_{i,t} = \boldsymbol{\mu}_{i,t+\Delta t} - \boldsymbol{\mu}_{i,t}\) across all primitives would erroneously forbid stretching, compression, or localized shearing. The proposed MCR penalizes only directional discrepancy relative to the group's mean displacement vector \(\bar{\mathbf{d}}_{\mathcal{N}_j,t} = \frac{1}{|\mathcal{N}_j|} \sum_{\mathcal{G}_i \in \mathcal{N}_j} \mathbf{d}_{i,t}\) via cosine similarity:

\[\mathcal{L}_{\mathrm{MCR}} = 1 - \frac{\mathbf{d}_{i,t} \cdot \bar{\mathbf{d}}_{\mathcal{N}_j,t}}{\lVert \mathbf{d}_{i,t} \rVert \lVert \bar{\mathbf{d}}_{\mathcal{N}_j,t} \rVert + \epsilon}\]

Applied only to active Gaussians satisfying \(\lVert \mathbf{d}_{i,t} \rVert > 0.0001\), MCR ensures that neighboring primitives move in coordinated directions while permitting spatially variable velocity magnitudes, cleanly avoiding the over-constraining rigidity trap.

3. Spectral Regularization (SR): Rotation-invariant shape preservation via covariance eigenvalue matching

While MCR guides motion directions, it does not explicitly prevent the group from collapsing or flying apart. Instead of adopting strict pairwise distance penalties (such as ARAP), the proposed spectral regularization maintains the overall geometric distribution of the group by comparing the eigenvalue spectra of 3D positional covariance matrices at time \(t\) and \(t+\Delta t\). Let \(\sigma_{t,r}\) and \(\sigma_{t+\Delta t,r}\) (\(r \in \{1, 2, 3\}\)) denote the ascending eigenvalues of covariance matrices \(\mathbf{K}_t\) and \(\mathbf{K}_{t+\Delta t}\). The consistency loss is formulated with a Huber loss:

\[\mathcal{L}_{\mathrm{SR}} = \sum_{r=1}^3 \mathrm{Huber}\bigl(\sigma_{t,r}, \sigma_{t+\Delta t,r}\bigr)\]

Because eigenvalues represent the principal variance lengths of the spatial cluster, they are invariant to 3D rigid rotations and translations. This loss constrains the group from undergoing unnatural spatial expansion or distortion, pulling diverging outlier Gaussians back to their coherent collective distribution while granting complete freedom to rotation and smooth non-rigid deformation.

4. Welford's Algorithm: Efficient single-pass online covariance calculation

Computing covariance matrices per ray in parallel without redundant passes or excessive GPU memory consumption is non-trivial. The framework incorporates Welford's online update algorithm. Given the running mean \(\bar{\boldsymbol{\mu}}_i = \frac{1}{i}\sum_{j=1}^i \boldsymbol{\mu}_j\), the covariance matrix \(\mathbf{K}_i\) is updated recursively upon integrating the \(i\)-th Gaussian:

\[\mathbf{K}_i = \frac{i-1}{i}\mathbf{K}_{i-1} + \frac{1}{i}\mathbf{P}_i, \quad \mathbf{P}_i := (\boldsymbol{\mu}_i - \bar{\boldsymbol{\mu}}_{i-1})(\boldsymbol{\mu}_i - \bar{\boldsymbol{\mu}}_i)^\top\]

This formulation enables the covariance of each ray-based group to be accumulated in a single pass directly during rasterization, making the spectral loss computationally practical for dense iterative training.

Loss & Training

The complete training objective combines the standard photometric \(L_1\) loss and structural dissimilarity loss \(\mathcal{L}_{\mathrm{dssim}}\) with the proposed motion and spectral terms:

\[\mathcal{L} = (1 - \lambda_{\mathrm{dssim}}) \mathcal{L}_1 + \lambda_{\mathrm{dssim}} \mathcal{L}_{\mathrm{dssim}} + \lambda_{\mathrm{MCR}} \mathcal{L}_{\mathrm{MCR}} + \lambda_{\mathrm{SR}} \mathcal{L}_{\mathrm{SR}}\]

The regularizers operate strictly during training. Because no architectural parameters, deformation networks, or rasterizer kernels are altered at test time, the model delivers zero-overhead real-time rendering.

Key Experimental Results

Main Results

Quantitative evaluations across synthetic (D-NeRF) and challenging real-world monocular benchmarks (HyperNeRF, NeRF-DS) demonstrate substantial and consistent rendering improvements across four distinct baseline architectures.

Dataset / Baseline Model	Metric System (PSNR↑ / SSIM↑ / LPIPS↓)	Baseline Performance	Baseline + Ours	Gain (\(\Delta\) PSNR)
D-NeRF (Ex4DGS)	PSNR / SSIM / LPIPS_V	29.93 / 0.968 / 0.041	31.04 / 0.972 / 0.038	+1.11 dB
D-NeRF (RTD)	PSNR / SSIM / LPIPS_V	35.36 / 0.985 / 0.022	36.46 / 0.987 / 0.019	+1.10 dB
D-NeRF (MoDec-GS)	PSNR / SSIM / LPIPS_V	29.33 / 0.963 / 0.048	31.68 / 0.974 / 0.035	+2.35 dB
D-NeRF (Grid4D)	PSNR / SSIM / LPIPS_V	42.00 / 0.994 / 0.008	42.20 / 0.994 / 0.008	+0.20 dB
HyperNeRF (Ex4DGS)	PSNR / MS-SSIM / LPIPS_A	24.61 / 0.829 / 0.199	24.86 / 0.836 / 0.233	+0.25 dB
HyperNeRF (RTD)	PSNR / MS-SSIM / LPIPS_A	25.17 / 0.841 / 0.283	25.30 / 0.845 / 0.273	+0.13 dB
HyperNeRF (MoDec-GS)	PSNR / MS-SSIM / LPIPS_A	24.92 / 0.841 / 0.176	25.09 / 0.846 / 0.173	+0.17 dB
HyperNeRF (Grid4D)	PSNR / MS-SSIM / LPIPS_A	25.50 / 0.783 / 0.185	25.56 / 0.856 / 0.198	+0.06 dB (MS-SSIM +0.073)
NeRF-DS (MoDec-GS)	PSNR / SSIM(MS) / LPIPS_V	22.93 / 0.822(0.851) / 0.229	23.76 / 0.848(0.885) / 0.184	+0.83 dB
NeRF-DS (Grid4D)	PSNR / SSIM(MS) / LPIPS_V	23.41 / 0.833(0.872) / 0.196	23.70 / 0.840(0.882) / 0.188	+0.32 dB

Ablation Study

Ablation experiments conducted on RTD across D-NeRF and HyperNeRF scrutinize grouping mechanisms (KNN vs Ray-based Grouping / RG) and loss formulations (ARAP, MCR, SR):

Grouping & Regularization Config	D-NeRF PSNR↑	D-NeRF SSIM↑	D-NeRF LPIPS_V↓	D-NeRF Training Time (s)	HyperNeRF PSNR↑	HyperNeRF MS-SSIM↑	HyperNeRF Training Time (s)
RTD Baseline (Unregularized)	35.36	0.9845	0.0224	841	25.17	0.8413	1963
RTD + KNN & ARAP	34.80	0.9828	0.0258	1484	23.18	0.7532	5808
RTD + KNN & Full (MCR+SR)	33.29	0.9771	0.0290	2044	25.11	0.8390	9586
RTD + RG & ARAP	35.27	0.9844	0.0216	2125	25.25	0.8430	13096
RTD + RG & MCR (Motion Only)	36.23	0.9866	0.0190	1649	24.23	0.8449	4454
RTD + RG & SR (Spectral Only)	34.98	0.9841	0.0220	1501	24.91	0.8331	6694
RTD + RG & Full (Proposed Full Model)	36.46	0.9869	0.0193	1924	25.30	0.8564	7138

Key Findings

• Failure of Euclidean KNN Grouping: Enforcing motion constraints over KNN clusters severely degrades performance on both datasets (e.g., HyperNeRF PSNR drops to 23.18 and MS-SSIM to 0.7532 with KNN & ARAP). KNN groups across occluded surfaces, dragging background and foreground together.
• Mutual Synergy between MCR and SR: MCR alone benefits linear motion in synthetic D-NeRF (35.36 \(\to\) 36.23) but deteriorates on real-world complex motion in HyperNeRF (25.17 \(\to\) 24.23) due to lack of shape constraints; SR alone cannot actively align velocities. Combining RG with both terms (Full) produces the best performance across all metrics.
• Training Efficiency over Spatial Queries: RG & Full achieves faster training than KNN & Full (7138s vs 9586s on HyperNeRF, saving over 25% training overhead) while adding zero runtime overhead during inference.

Highlights & Insights

• Repurposing Volume Rendering as a Geometric Clusterer: Instead of maintaining external spatial index trees or KD-trees, treating pixel viewing rays and accumulated alpha weights as a free, occlusion-aware grouping operator bridges 2D perspective observations and 3D surface continuity.
• Distributional Spectral Regularization over Rigid Distance Locks: Shifting from rigid Euclidean distances to eigenvalue spectrum matching yields rotation invariance and non-rigid elasticity while preserving macroscopic volume and shape, eliminating boundary floaters and distortion.
• Pure Self-Supervised Physical Coherence: The framework achieves state-of-the-art temporal consistency entirely from photometric optimization without optical flow, depth estimators, or multi-view cameras, demonstrating the power of intrinsic geometric priors.

Limitations & Future Work

• Training Time Overhead: Due to the online covariance calculation and SVD operations required at each optimization step, training time increases by approximately \(1.5\times\) to \(2\times\) relative to unregularized baselines.
• Sensitivity to Fixed Visibility Threshold \(\tau\): In regions with delicate volumetric phenomena (e.g., thin smoke, fine hair strands, transparent glass), a constant alpha threshold may inadvertently omit sparsely blended primitives that contribute subtle dynamic structure.
• Incorporating Higher-Order Kinematics: The formulation presently constrains first-order displacements and positional covariances across adjacent frames; extending this to acceleration and physical inertia promises even greater fidelity under erratic motion.

Related Work & Insights

• vs SC-GS / Dynamic 3D Gaussians (Rigidity via KNN): Prior works enforce ARAP or rigid transforms over KNN neighborhoods. This paper reveals that KNN induces cross-surface contamination and ARAP restricts non-rigid elasticity; replacing them with ray grouping and spectral loss resolves both flaws.
• vs MotionGS / Shape of Motion (External 2D/3D Priors): Prior approaches rely heavily on optical flow (RAFT/GMFlow) or tracking models (CoTracker). This method operates entirely prior-free, remaining resilient in specular (NeRF-DS) and challenging monocular scenes where 2D motion trackers break down.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Ingenious repurposing of ray alpha blending for grouping and eigenvalue matching for relaxed rigidity)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated on 4 diverse baselines, 3 benchmarks, with comprehensive ablations and trajectory analyses)
• Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear theoretical motivation, and self-consistent derivations)
• Value: ⭐⭐⭐⭐⭐ (Model-agnostic, prior-free, zero inference cost; establishes a practical regularizer for dynamic 3DGS)