Skip to content

SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision

Conference: ECCV 2026
arXiv: 2603.24036
Code: Not mentioned (1D/2D demo code included in supplementary material)
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Differentiable Tracking, Frequency Domain Supervision, Frequency Annealing, Basin of Attraction

TL;DR

This work shifts the supervision target of 3DGS differentiable tracking from the spatial domain (pixel-wise photometric error) to the frequency domain (spectral moments). Combined with a frequency annealing schedule strictly derived from the principle of phase wrapping, the method enables Gaussian assets to obtain non-zero directional gradients even when completely non-overlapping with the target at initialization, smoothly "flowing" to the correct pose—thereby enabling robust tracking without manual alignment or category priors.

Background & Motivation

3D Gaussian Splatting (3DGS) has rapidly rewritten the landscape of 3D reconstruction with its real-time, photo-realistic novel view synthesis. A natural downstream application is to animate the reconstructed static Gaussian assets—fitting them to a target video through a differentiable renderer to enable tasks like digital avatar driving, markerless motion capture, and editable dynamic scene reconstruction. Such continuous model-based tracking essentially optimizes motion parameters \(\Theta\) to minimize the photometric error between rendered images and target observations. However, once leaving controlled laboratory settings for wild data, this seemingly straightforward path becomes notoriously fragile.

The root of the problem lies in the compact support and local nature of Gaussian primitives. Standard photometric objectives implicitly rely on spatial overlap: for a Gaussian to receive gradients pointing toward the target structure, its rendered footprint must already intersect with the target structure's location. When the initial pose is far from the target (due to coarse initialization, noisy pose priors, or large frame-to-frame displacements from fast motion), the rendered Gaussians do not overlap with the target pixels at all, causing the gradient components pointing to the ground-truth target to strictly vanish. Worse, the total gradient of the scene is not zero; misplaced Gaussians inevitably overlap with background clutter, feeding "corrupted gradients" to the optimizer. Instead of pulling the object toward the target, this anchors it firmly to the background. The authors clearly dissect this "vanishing gradient" pathology in 1D: under large displacement, the standard spatial \(L_2\) landscape has no global basin of attraction leading to the correct solution, resulting in catastrophic tracking failure. Existing methods either rely on manual alignment or controlled capture to ensure sufficient overlap in the first frame, or, as in recent works, replace \(L_2\) with deep feature distances like LPIPS. The hierarchical receptive fields of the latter expand the basin of attraction slightly, but still fundamentally rely on local overlap; once the asset and the target are disjoint, the gradient still vanishes. Other methods rely on category-specific priors like SMPL or articulated templates, using off-the-shelf pose estimators for strong initial alignment and treating photometric tracking as a "last-mile" fine-tuning step—at the cost of sacrificing generality and failing to track arbitrary in-the-wild objects. This leaves an open gap: a purely optimization-based tracking objective that is both global (robust to large, non-overlapping displacements) and category-agnostic.

The key insight of this paper is: since pixels and rendered Gaussians are local, one should switch to a set of global basis functions. The sinusoidal basis spans the entire spatial domain, projecting the rendered image onto a set of complex Fourier features to obtain a "spectral signature" of the current pose. Spatial translations of the object correspond to phase shifts in the frequency domain—providing strong, non-zero directional gradients even when the object and the target are completely disjoint in space. Core Idea: Shift the tracking supervision target from the spatial domain to the frequency domain, aligning the rendered image and the target using a set of global complex sinusoidal features (spectral moments) to construct a global basin of attraction across the entire image space; then, use a frequency annealing schedule strictly derived from the phase-wrapping condition \(|\omega^T d|<\pi\) to smoothly transition the optimizer from "low-frequency global convexity" to "high-frequency precise alignment", obtaining both non-vanishing frequency-domain gradients and restoring spatial accuracy.

Method

Overall Architecture

SpectralSplats is a model-agnostic tracking framework. Given a static reference Gaussian model \(\mathcal{G}_{\text{ref}}\) and a target video, it solves for a set of motion parameters \(\Theta\) that drive a deformation function \(\mathcal{D}\) to shape and align the rendered images with the target. It does not modify any underlying deformation models, but simply replaces the first phase of the training objective from spatial photometric loss to spectral moment loss. The optimization proceeds in two phases: first, use a progressively annealed spectral objective to establish a global basin of attraction, rescuing Gaussians from zero-overlap "stranded" states to resolve severe initial misalignment; once the spectral phase pulls the object to local overlap, seamlessly switch to standard spatial loss (pixel L2 or LPIPS) via Parseval's theorem for high-frequency refinement.

The method divides its contributions into three parts: first diagnosing the mathematical root of the "vanishing gradient/locality trap" failure mode; then replacing pixel-wise comparisons with global projections using spectral moments, converting spatial translations into phase shifts via the Fourier shift theorem to obtain non-zero gradients; and finally deriving a frequency annealing schedule from phase-wrapping analysis to dynamically control the bandwidth of active frequencies, avoiding high-frequency spurious local minima. The entire pipeline is as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Canonical Gaussians G_ref<br/>+ Target Video I_gt"] --> B["Deformation + Rasterization<br/>Render Current Pose I_rend"]
    B --> C["Spectral Moment Supervision<br/>Project onto Complex Sinusoidal Basis<br/>Establish Global Basin of Attraction"]
    C --> D["Frequency Annealing Schedule<br/>Under Phase-Wrapping Constraints<br/>Gradually Activate Low to High Frequencies"]
    D -->|Spectral phase pulls to local overlap| E["Switch to Spatial Loss<br/>Pixel L2/LPIPS High-frequency Refinement"]
    E --> F["Output Aligned Pose<br/>+ Frame-by-frame Deformation"]

Key Designs

1. Spectral Moment Loss: Rooting out "vanishing gradients" with global sinusoidal projection

The pain point is straightforward: Gaussians are compactly supported local primitives. Pixel-wise photometric losses require spatial overlap with the target object to yield directional gradients; at zero overlap, the gradient is strictly zero, leaving the optimizer stranded. The authors mathematically formalize this trap by decomposing the gradient of the squared error for a source-target pair into a "self-term" and a "target-supervised cross-term": in-plane translations keep the overall footprint mass of the rendered object invariant, leaving the self-term strictly invariant under motion parameters (its derivative is exactly zero). (While depth translations yield non-zero derivatives, under zero overlap they only drive the object to shrink and move away from the camera, failing to provide directional signals toward the target). The only component carrying tracking signals is the target-supervised cross-term. When the rendered Gaussians do not overlap with the ground-truth target position, the multiplication of \(\mathbf{I}_{\text{gt}}(\mathbf{p})\) with the spatial boundary of the rendered object \(\nabla_\Theta\mathbf{I}_{\text{rend}}\) is zero everywhere, causing the entire term to vanish. This trap is further exacerbated by the 3DGS architecture itself—for real-time performance, the rasterizer tiles the screen into \(16\times16\) blocks and discards primitives outside a 99% confidence interval, directly zeroing out gradients for targets outside the tile neighborhood.

The solution is to replace pixel-wise comparisons with image moment alignment: a moment is equivalent to multiplying the image by an auxiliary static field \(F(\mathbf{p})\) and integrating. As long as a field that varies continuously across the spatial domain without repeating values (like a sine wave or polynomial) is chosen, this projection acts as a global coordinate system, breaking the locality trap. Specifically, the authors select complex sinusoids as the fields—since they exhibit geometrically meaningful phase shifts under translation—defining the spectral moment at a 2D spatial frequency \(\omega_{k_x,k_y}\) as:

\[\mathcal{M}(k_x,k_y;\mathbf{I})=\sum_{\mathbf{p}}\mathbf{I}(\mathbf{p})\cdot\exp(-j\,\omega_{k_x,k_y}^{T}\mathbf{p})\]

The gradient of the moment matching loss \(\mathcal{L}_{\text{moment}}=\tfrac12(M_{\text{rend}}-M_{\text{gt}})^2\) consists of two reliably non-zero components: as long as the global field does not repeat its values spatially, the scalar projections of the non-overlapping rendered object and target will inevitably differ, ensuring a non-zero error magnitude; the directional component \(\nabla_\Theta M_{\text{rend}}\) can be written via the chain rule and integration by parts as \(\int\mathbf{I}_{\text{rend}}(\mathbf{p};\Theta)\nabla_{\mathbf{p}}F(\mathbf{p})\,d\mathbf{p}\). As long as the spatial derivative of the field is non-zero at the object's current location, the optimizer "senses" the slope of the field. In short: the scalar difference determines the pull magnitude, and the field gradient determines the pull direction, enabling registration without correspondences even when the source and target are completely disjoint. This formulation natively leverages 2D FFT for efficient computation, avoiding the prohibitive cost of naive evaluation over dense frequency grids.

2. Frequency Annealing Schedule: Deriving "low-to-high" safe expansion rate from phase-wrapping conditions

Using spectral moments alone is insufficient, as a fundamental paradox arises: by Parseval's theorem, optimizing over the entire orthogonal frequency basis simultaneously makes the spectral loss strictly equivalent to the spatial \(L_2\) loss. Thus, opening all frequencies at once simply restores the vanishing gradients and local minima of the spatial domain. High-frequency components introduce severe phase wrapping, fragmenting the global basin into spurious local minima and trapping the optimizer. Rather than statically utilizing the entire basis, the active frequency band must be dynamically controlled during optimization: low frequencies provide global attraction but lack precision (the spatial gradient of the spectral loss scales with frequency magnitude, \(\nabla\mathcal{L}\propto\omega\sin(\omega d)\), decaying as the spatial error approaches zero), while high frequencies offer precision but cause phase wrapping if activated too early. Thus, a systematic coarse-to-fine transition is required.

Crucially, the authors do not heuristically devise the annealing rate but derive it from first principles. For a given frequency \(\omega\), the spectral loss is convex only when the induced phase shift is not wrapped, i.e., \(|\omega^T\mathbf{d}_t|<\pi\). In the supplementary material, the Fourier shift theorem is used to simplify the loss to \(E(\mathbf{d};\omega)=|\mathcal{M}_{\text{gt}}|^2(1-\cos(\omega^T\mathbf{d}))\). Its stationary points satisfy \(\sin(\omega^T\mathbf{d})=0\), meaning \(\mathbf{d}=0\) is the unique attractor only within \(|\omega^T\mathbf{d}|<\pi\), beyond which it falls into spurious basins of subsequent cycles. This phase-wrapping condition dictates a dynamic stability constraint: the maximum active frequency magnitude must be inversely proportional to the spatial error magnitude, \(\|\omega_{\max}(t)\|\propto 1/\|\mathbf{d}_t\|\). Within the strongly convex region satisfying this constraint, the loss approximates a quadratic bowl where the gradient is proportional to displacement, and gradient descent naturally scales its step size by the remaining distance, leading to exponential decay of the spatial error: \(\|\mathbf{d}_t\|\le\|\mathbf{d}_0\|\gamma^t\). To maintain this constraint, the active frequency magnitude must expand exponentially as \(\gamma^{-t}\). Since the frequencies of a standard spectral grid are logarithmically spaced, \(\|\omega_k\|\propto 2^k\), an exponentially growing frequency magnitude corresponds to an active frequency index \(k(t)\) expanding strictly linearly with the iteration count. This derivation conversely provides a rigorous first-principles explanation for the empirical success of linear annealing schedules in Nerfies and BARF, which previously relied on heuristic arguments about NTK or signal bandwidth; a linear index is indeed the theoretical upper bound for safe frequency expansion. In practice, a smooth cosine weight is used to gradually fade in higher frequency bands:

\[w_k(t)=\frac{1-\cos(\pi\cdot\text{clamp}(\alpha(t)-k,0,1))}{2}\]

where \(\alpha(t)\) linearly increases from 0 to \(K\) over iterations to control the active bandwidth.

3. Conservative Frequency Expansion + Model-Agnostic Two-Phase Transition: Bringing theory to in-the-wild tracking

The above derivation assumes ideal linear convergence, but in real tracking scenarios, background clutter, occlusions, and complex deformations often disrupt ideal exponential error decay. The authors therefore implement a double-conservative schedule: first, following BARF's empirical practice, they enforce a strict low-frequency warm-up phase, freezing \(\alpha(t)\) for the first few iterations to allow the optimizer to resolve severe initial misalignments using the widest global basin before introducing high-frequency complexity (though the warm-up cannot be too long, or high-frequency details cannot be restored). Second, once expansion begins, they scale the frequency itself linearly instead of its logarithmic index—since linear growth is much slower than exponential, this practical relaxation keeps the optimization safely below the phase-wrapping threshold \(|\omega^T\mathbf{d}_t|<\pi\). This delayed, sub-exponential expansion significantly improves robustness. Additionally, the framework operates solely on foreground Gaussians. Foreground masks are extracted once per scene using standard 3D segmentation tools, and the background is recombined during rendering.

The framework is model-agnostic because the spectral loss is applied purely to the rendered outputs, making no assumptions about the underlying deformation model. The authors validate this on two mainstream non-rigid parameterizations: one using a neural MLP like TimeNet to continuously predict deformations for sparse control points, and another directly optimizing control point displacements and rotations (direct morph field). Spectral supervision coaxes these deformation models into high-accuracy final poses from extreme initial displacements where standard photometric loss would fail, without modifying the deformation models themselves. The transition to photometric loss is also natural. Once the spectral phase pulls the object to local overlap, the training seamlessly switches to spatial loss (or LPIPS for the synthetic SC4D dataset) for high-frequency refinement based on Parseval equivalence, while still incorporating the As-Rig-As-Possible (ARAP) regularization utilized in GSGD to encourage locally rigid motion of the control points.

Loss & Training

Two-phase optimization. In the spectral phase, the image loss computes the L1 difference of spectral moments for RGB and opacity respectively over the active frequency band \(\mathcal{K}(t)\) with weights \(w_k(t)\) (with a weighted mask term \(\lambda_{\text{mask}}\)). In the spatial phase, it transitions to pixel L2 + mask L2 + opacity BCE (optionally LPIPS for SC4D). The overall loss is formulated as \(\mathcal{L}=\lambda_{\text{image}}\mathcal{L}_{\text{image}}+\lambda_{\text{arap}}\mathcal{E}_{\text{arap}}\), where \(\lambda_{\text{image}}\) is fixed at 5000 and ARAP is introduced from step 1000. All experiments employ 800 control points and train for 10K steps. The spectral-to-spatial transition step (add_pixel_loss), the number of frequency bands \(K\), and the warm-up ratio are tuned per dataset (e.g., for GART: transition step 7000, \(K{=}8\), warm-up ratio 0.25). Optimization takes approximately 8–15 minutes per sequence on a single L40 GPU, introducing almost zero extra overhead compared to pixel loss (437s vs 443s on SC4D, 0-6% overhead across experiments).

Key Experimental Results

Main Results

Evaluated on two datasets: SC4D (clean and controllable 4D animations generated from Consistent4D assets, with well-aligned initial appearance and supervision) and GART Dog (rest pose models reconstructed from real monocular videos, featuring illumination inconsistency, unknown camera views, and significant pose/appearance discrepancies). Misalignment is simulated by shifting the initial 3DGS model along random directions with increasing translation offsets. The core comparison evaluates the spectral approach (+Ours) stacked onto different deformation parameterizations and spatial losses, against pure spatial supervision (Pixel).

GART (shift = 0.6) main results, average performance (higher is better / lower is better for LPIPS):

Dataset Metric Ours Pixel DT PoG
GART Dog PSNR↑ 22.06 20.15 20.92 17.09
GART Dog SSIM↑ 0.907 0.892 0.900 0.863
GART Dog LPIPS↓ 0.217 0.259 0.226 0.277

As shown, the proposed method not only consistently outperforms the pixel-wise baseline but also comprehensively beats two global loss baselines (Euclidean Distance Transform DT and Pyramid of Gaussians PoG)—both of which tend to merge multiple separate objects during optimization (such as pulling the strawberry into the banana in the teaser scene), while the proposed method always keeps objects separate.

SC4D (shift = 0.5) cross-parameterization comparison (training views, MLP w/o LPIPS configuration):

Configuration PSNR↑ SSIM↑ LPIPS↓
MLP + Pixel 17.67 0.911 0.181
MLP + Ours 26.70 0.951 0.052

The larger the misalignment, the wider the gap between pixel supervision and the proposed method. This holds true on both training and novel views, indicating better generalization. Even with multi-view supervision, pixel loss still fails under spatial misalignment, whereas the proposed method remains robust.

Ablation Study

Configuration Key Metric Description
K = 4 / 6 / 8 / 10 / 12 PSNR 21.79 / 22.02 / 22.06 / 21.28 / 21.35 Sensitivity of number of frequency bands on GART: stable in a wide range, K=8 is optimal
Aligned initialization shift = 0.0 (MLP w/o LPIPS) PSNR 28.34 vs Pixel 23.72 The proposed method does not degrade when there is no misalignment, and performs even better in most cases
Spatial phase loss components (GART Shiba) MSE \(\rightarrow\) +Masked \(\rightarrow\) +BCE \(\rightarrow\) All: 20.65 \(\rightarrow\) 20.21 \(\rightarrow\) 20.55 \(\rightarrow\) 22.06 All components contribute to refinement; complete configuration is optimal

Key Findings

  • The frequency annealing schedule is critical for global convergence: static high-frequency bases prevent global gradient vanishing but trap the optimizer in spurious local minima due to phase wrapping. Only starting from low frequencies \(\rightarrow\) linear expansion can rescue zero-overlapping Gaussians without introducing spurious minima.
  • The more severe the misalignment, the larger the performance gain of the proposed method over the pixel baseline. This trend is consistent across PSNR/SSIM/LPIPS and multi-view setups—directly demonstrating the achievement of the "expanding the basin of attraction" goal.
  • LPIPS actually performs worse on GART: because LPIPS is sensitive to perceived color and brightness differences, it misinterprets global color discrepancies between the reconstructed 3DGS and the video as structural changes. Consequently, its gradients optimize for lighting disparities rather than forcing geometric consistency, leading to blurrier results—explaining why pixel L2 is more suitable for the second stage on real-world data.
  • The method is robust to the annealing schedule parameters within a wide range and introduces almost zero computational overhead compared to pixel loss, making it a drop-in loss replacement.

Highlights & Insights

  • Reframe the widely evaded engineering issue of "vanishing gradients" as a strictly analyzable optimization landscape problem: decomposing the gradient into self-terms and cross-terms to prove that plane translation yields exactly zero spatial gradients makes the analysis of "why it fails" clean and thorough, elevated beyond simple empirical statements like "heuristically not working."
  • Elegantly leverage frequency-spatial duality: Parseval's theorem acts both as a "warning" (preventing static optimizing over all bases, which renders spectral loss equivalent to \(L_2\)) and as a "permit" to seamlessly switch back to spatial loss once the spectral phase converges, utilizing one theorem for two complementary purposes.
  • Derive the phase-wrapping condition \(|\omega^T d|<\pi\) to show that "linear index expansion on a logarithmic grid is the theoretical upper bound for safe frequency expansion." This elegantly solidifies the empirical success of linear annealing schedules from BARF/Nerfies with first-principles proof, turning a heuristic practice into a proven theorem.
  • The spectral loss operates entirely on the rendered outputs, making zero assumptions about the deformation model. It can thus be utilized as a drop-in replacement compatible with various combinations like MLP, direct displacement fields, pixel L2, and LPIPS—this orthogonality presents great potential for transfer to any differentiable tracking pipeline relying on photometric targets.

Limitations & Future Work

  • The authors acknowledge that the current deformation assumption requires a pre-initialized canonical asset, restricting it to "model-based tracking" rather than covering full dynamic scene reconstruction where canonical geometry and motion must be jointly optimized from uncalibrated videos—representing the most natural extension.
  • Only sinusoidal (Fourier) moments are utilized. While the authors mention exploring other moment types to capture more complex dynamics as a worthwhile direction, the trade-offs of different global kernels (geometric moments, orthogonal polynomial moments) are not fully elaborated.
  • The method works exclusively on foreground Gaussians and relies on a one-time 3D segmentation mask per scene. In the basketball experiment, the dataset's masks contain flaws, and the performance boundaries under noisy masks are not systematically quantified.
  • The experimental scale is relatively small (a few characters in SC4D + 7 dogs in GART + 1 basketball sequence), and misalignments are synthetically generated via translations, raising questions about whether they match the distribution of real-world tracking failures. Joint large rotational and translational misalignments are only demonstrated in 2D, lacking quantitative evaluation on 3D data.
  • vs LPIPS deep feature tracking (e.g., GSGD, Bekor, etc.): They rely on the hierarchical receptive fields of deep features to widen the basin of attraction, but still depend on local spatial overlap; the gradient still vanishes at zero overlap. In contrast, the proposed method shifts the supervision to the frequency domain to obtain truly global, non-vanishing gradients, enabling convergence even under complete disjointness. Empirically, LPIPS performs worse on real-world scenes with illumination discrepancies.
  • vs Category prior methods (e.g., SMPL, HUGS, GART articulated templates): They employ off-the-shelf pose estimators for strong initial alignment, reducing the problem to "last-mile" refinement. This is robust but sacrifices generality, as it can only track known categories. The proposed method is purely optimization-based and category-agnostic, capable of tracking any in-the-wild object, and is complementary to these motion models (providing global supervision signals for them).
  • vs BARF / Nerfies frequency annealing: They perform spectral annealing on positional encodings in NeRF to expand the camera registration basin of attraction, justifying it heuristically via NTK or signal bandwidth. The proposed method applies the annealing directly on the spectral moments of the rendered outputs (rather than positional encodings) and provides a first-principles derivation and theoretical upper bound based on phase-wrapping analysis, while avoiding high-frequency phase-wrapping traps.
  • vs Representation quality-oriented frequency methods (e.g., FreGS, PGDGS, SAPE, Laplacian subbands): They use frequency decomposition primarily for level-of-detail control and static reconstruction fidelity. The proposed method utilizes frequency to reshape the geometric optimization landscape and resolve vanishing tracking gradients—differing entirely in objective.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Moving tracking supervision from space to frequency to resolve vanishing gradients, and deriving the annealing schedule strictly from phase-wrapping, presents a fresh perspective and solid theory.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Complete evaluation across two datasets, two parameterizations, two spatial losses, multi-views, and thorough ablations; however, the scale is relatively small and misalignments are synthetically generated.
  • Writing Quality: ⭐⭐⭐⭐⭐ Structured, presenting failure mode analysis, frequency duality, and annealing derivations progressively with clear figure-text consistency.
  • Value: ⭐⭐⭐⭐ Drop-in, near-zero overhead replacement, from which any differentiable tracking pipeline dependent on photometric objectives can benefit.