SkipGS: Post-Densification Backward Skipping for Efficient 3DGS Training

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ASU-ESIC-FAN-Lab/SkipGS
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Training Acceleration, Backward Skipping, Post-Densification, Novel View Synthesis

TL;DR

SkipGS observes that the backward pass accounts for over 60% of per-iteration runtime in the post-densification phase of 3D Gaussian Splatting (3DGS) while yielding diminishing gradient returns on converged views; by introducing a view-adaptive backward gating mechanism with an auto-calibrated cumulative budget controller, it cuts post-densification training time by 42.0% and end-to-end time by 23.1% on Mip-NeRF 360 with virtually identical rendering quality.

Background & Motivation

3D Gaussian Splatting (3DGS) has rapidly emerged as the leading standard for real-time novel view synthesis by rendering millions of learnable anisotropic 3D Gaussians through a fully differentiable tile-based rasterizer. Despite delivering high visual fidelity and interactive rendering frame rates, training remains computationally demanding: standard pipelines routinely require tens of thousands of iterations, posing a substantial practical bottleneck for interactive scene reconstruction, large-scale benchmarks, and automated downstream processing. To accelerate training, extensive research has focused almost exclusively on the "primitive dimension"—either compressing Gaussian counts via training-time or post-hoc pruning (e.g., FastGS, LightGaussian) or constraining Gaussian growth during adaptive densification under compute budgets (e.g., Taming 3DGS). These works primarily optimize the workload processed per iteration by adjusting how many Gaussians participate in rendering.

However, existing approaches leave largely unexploited an orthogonal axis rooted in the intrinsic temporal phase transition of 3DGS training. Specifically, 3DGS optimization naturally splits into an early densification phase (typically iterations 0 to 15k), where the Gaussian set expands aggressively to reconstruct complex scene geometry, and a subsequent post-densification phase (iterations 15k to 30k), where the primitive count is frozen and optimization transitions into parameter refinement. Runtime and gradient profiling reveal that once densification halts, the backward pass dominates the per-iteration cost (accounting for approximately 62% of total runtime); concurrently, per-Gaussian gradient norms drop roughly twofold from early stages and flatten completely after densification stops. Meanwhile, Adam update norms remain artificially elevated due to momentum inertia, indicating that numerous post-densification backward passes compute weakly informative gradients that barely move the scene parameters. Nevertheless, standard 3DGS continues to execute full backpropagation unconditionally for every sampled camera view.

Because different training views converge at vastly different rates depending on visibility, depth complexity, and view-dependent effects, treating all sampled viewpoints uniformly in late training produces massive computational waste. The core idea is to formulate post-densification 3DGS training as a constrained backward gating problem: by always executing lightweight forward passes to dynamically track per-view loss baselines, SkipGS selectively skips redundant backward passes whenever a view's loss aligns with its recent baseline, while enforcing an auto-calibrated minimum cumulative backward budget to prevent gradient starvation and guarantee stable convergence.

Method

Overall Architecture

SkipGS operates specifically during the post-densification phase of 3DGS training (from \(T_d = 15\text{k}\) to \(T = 30\text{k}\) iterations), where the Gaussian primitive set remains fixed. The overall pipeline consists of an initial warmup calibration period followed by online constrained backward gating. In every iteration, SkipGS samples a training view and unconditionally executes the forward pass to evaluate photometric reconstruction loss and update that view's exponential moving average (EMA) baseline. Next, it computes a normalized deviation score measuring whether the current view's loss represents an unexpected error spike. If the deviation score does not exceed 1, the gate proposes skipping the backward pass. Before skipping, a cumulative budget controller checks whether the global backward execution ratio has dropped below an auto-calibrated safety threshold; if so, backpropagation is enforced. Otherwise, the backward pass and Adam optimizer step are omitted, moving directly to the next iteration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Sample view v_t & run forward pass<br/>Compute photometric loss L(t)"] --> B["Per-view deviation score<br/>Compare current loss to baseline yielding s"]
    B --> C["Skip test & decoupled baseline update<br/>s > 1 proposes backward, s <= 1 proposes skip; update EMA"]
    C --> D{"Propose backward skip<br/>(s <= 1) ?"}
    D -->|No: s > 1| F["Execute backward pass & Adam step<br/>Increment backward counter b = b + 1"]
    D -->|Yes: propose skip| E{"Warmup calibration & cumulative budget control<br/>Cumulative ratio rho_cum < rho_min ?"}
    E -->|Yes: trigger budget override| F
    E -->|No: allow skipping| G["Skip backward pass & optimizer step<br/>Only incur forward runtime, proceed to next iter"]
    F --> H["Proceed to next iteration (t = t + 1)"]
    G --> H

Key Designs

1. Per-view deviation score: tracking convergence heterogeneity via independent EMA baselines In post-densification training, scene-wide loss curves obscure the fact that simple, unobstructed viewpoints converge far earlier than complex, disoccluded, or specular regions. Imposing a fixed global loss threshold would cause severe under-fitting in difficult views or redundant updates on simple ones. SkipGS maintains an independent exponential moving average (EMA) loss baseline \(\bar{\mathcal{L}}_v\) for every training viewpoint \(v\). When view \(v_t\) is sampled at iteration \(t\), SkipGS computes a scale-invariant, normalized deviation score: $\(s_{v_t}^{(t)} = \frac{\mathcal{L}_{v_t}^{(t)}}{\bar{\mathcal{L}}_{v_t}^{(t-1)} + \epsilon}\)$ where \(\epsilon = 10^{-8}\) prevents division by zero. If a view is sampled for the very first time, the score defaults to \(s \leftarrow +\infty\) to ensure backward execution. A deviation score significantly exceeding 1 indicates that the current rendering error has spiked relative to its recent track record (e.g., due to coupled updates from adjacent views or unoptimized high-frequency details), signaling high gradient utility. Conversely, a score \(s \le 1\) demonstrates that the current view is already performing at or above its historical baseline, meaning a full backward pass yields diminishing returns.

2. Skip test and decoupled baseline update: forward tracking without backward overhead Conventional sample-skipping techniques often bypass entire iterations, leaving the model blind to whether skipped samples have degraded as neighboring primitives shifted. SkipGS decouples forward tracking from backward execution: forward rasterization is computationally inexpensive (accounting for only ~38% of per-iteration runtime), while backpropagation and atomic gradient accumulation consume over 62% of the budget. The skip decision is formulated as: $\(g_t(v_t) = \mathbb{I}\left[s_{v_t}^{(t)} > 1\right]\)$ Crucially, regardless of whether \(g_t(v_t)\) triggers backpropagation, SkipGS unconditionally updates the sampled view's EMA baseline immediately after loss computation: $\(\bar{\mathcal{L}}_{v_t}^{(t)} = \beta \bar{\mathcal{L}}_{v_t}^{(t-1)} + (1 - \beta) \mathcal{L}_{v_t}^{(t)}\)$ with \(\beta = 0.95\). This continuous updating ensures the baseline accurately mirrors the evolving capabilities of the 3D scene representation, preventing historical baselines from drifting out of date when views are repeatedly skipped.

3. Warmup calibration and cumulative budget control: a safety floor against gradient starvation Unconstrained skipping based purely on instantaneous loss ratios is fragile: if multiple views simultaneously satisfy \(s \le 1\), the optimizer risks prolonged gradient starvation, causing primitive parameters to stall or Adam momentum buffers to deplete prematurely. SkipGS erects a dual-layer safeguard. First, during an initial warmup period of \(W = 500\) iterations immediately following densification, all sampled views execute backward unconditionally while the system records the hypothetical trigger rate: $\(\hat{\rho}_W = \frac{1}{|\mathcal{T}_W|} \sum_{i \in \mathcal{T}_W} g_i(v_i)\)$ over warmup steps \(\mathcal{T}_W\) with existing EMA histories. At iteration \(t = W + 1\), the minimum required backward budget is automatically calibrated via linear interpolation: $\(\rho_{\min} = \rho_{\mathrm{lo}} + (1 - \rho_{\mathrm{lo}}) \hat{\rho}_W\)$ with baseline floor \(\rho_{\mathrm{lo}} = 0.5\). During subsequent gating, SkipGS tracks the cumulative backward execution ratio \(\rho_{\mathrm{cum}}(t-1) = b / \max(t-1, 1)\). If \(\rho_{\mathrm{cum}}(t-1) < \rho_{\min}\), the budget controller overrides the skip decision (\(g_t \leftarrow 1\)) and forces backward execution, ensuring that scene-adaptive minimum gradient flow is rigorously maintained without manual hyperparameter tuning.

A Worked Example

Consider an optimization run transitioning out of densification at \(T_d = 15\text{k}\): 1. Warmup phase (\(t = 1 \dots 500\)): Both forward and backward passes execute for every iteration. In the background, SkipGS observes that only 30% of iterations exhibit loss spikes (\(s > 1\)), giving \(\hat{\rho}_W = 0.30\). At iteration 501, the minimum budget is calibrated to \(\rho_{\min} = 0.5 + (1 - 0.5) \times 0.30 = 0.65\) (mandating that at least 65% of post-densification steps perform backpropagation). 2. Normal skipping: At iteration 2,000, view \(A\) is sampled with historical baseline \(\bar{\mathcal{L}}_A = 0.0420\). The forward pass computes \(\mathcal{L}_A^{(2000)} = 0.0415\). The deviation score is \(s = 0.0415 / 0.0420 = 0.988 \le 1\), proposing a skip. The budget controller checks \(\rho_{\mathrm{cum}} = 0.68 > 0.65\), which satisfies the threshold; hence the backward pass and optimizer update are omitted, updating only \(\bar{\mathcal{L}}_A\) and saving over 60% of iteration time. 3. Budget override: At iteration 8,000, view \(B\) is sampled and records \(s = 0.97 \le 1\). However, following several consecutive skips, the cumulative backward ratio has slipped to \(\rho_{\mathrm{cum}} = 0.648 < 0.65\). The budget controller immediately forces an override (\(g \leftarrow 1\)), executing full backpropagation and incrementing \(b \leftarrow b + 1\) to restore the budget to a safe trajectory.

Loss & Training

SkipGS maintains the standard 3DGS optimization objective and loss formulation without alterations: $\(\mathcal{L} = (1 - \lambda) \mathcal{L}_1 + \lambda (1 - \text{SSIM})\)$ with \(\lambda = 0.2\). When backpropagation is gated active, gradients flow back through tile-based rasterization and covariance projections to update Gaussian positions, rotations, scales, opacities, and spherical harmonic coefficients via Adam. When skipped, all primitive states remain untouched. All benchmarks were run on a single NVIDIA RTX Ada 5000 (32GB) GPU with default parameters: \(W = 500\), \(\beta = 0.95\), and \(\rho_{\mathrm{lo}} = 0.5\) across all scenes and datasets without per-dataset tuning.

Key Experimental Results

Main Results

SkipGS was evaluated across three standard real-world benchmarks: Mip-NeRF 360, Deep Blending, and Tanks & Temples. It was paired as a plug-in with Vanilla 3DGS and five leading efficiency-oriented baselines spanning Gaussian compaction, sparsification, and growth regulation. The table below reports average rendering quality metrics, total wall-clock training time (\(T_{\text{total}}\)), and post-densification refinement time (\(T_{\text{post}}\)) on Mip-NeRF 360:

Dataset / Method	Setting	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	\(T_{\text{total}}\) (s) ↓	\(T_{\text{post}}\) (s) ↓
Mip-NeRF 360 (Vanilla 3DGS)	Baseline	27.52	0.816	0.215	1705.7	939.6
Mip-NeRF 360 (Vanilla 3DGS)	+ SkipGS (Ours)	27.52	0.816	0.217	1311.1 (-23.1%)	545.0 (-42.0%)
Mip-NeRF 360 (FastGS)	Baseline	27.56	0.798	0.261	181.9	87.2
Mip-NeRF 360 (FastGS)	+ SkipGS (Ours)	27.51	0.797	0.262	164.5 (-9.6%)	69.8 (-20.0%)
Mip-NeRF 360 (Taming 3DGS)	Baseline	27.94	0.822	0.207	1339.0	757.0
Mip-NeRF 360 (Taming 3DGS)	+ SkipGS (Ours)	27.92	0.822	0.209	974.0 (-27.3%)	392.0 (-48.2%)
Mip-NeRF 360 (GaussianSpa)	Baseline	27.61	0.826	0.213	2640.9	1485.0
Mip-NeRF 360 (GaussianSpa)	+ SkipGS (Ours)	27.60	0.825	0.215	2490.9 (-5.7%)	1335.0 (-10.1%)
Mip-NeRF 360 (LightGaussian)	Baseline	27.49	0.810	0.230	240.0	240.0
Mip-NeRF 360 (LightGaussian)	+ SkipGS (Ours)	27.46	0.809	0.231	204.8 (-14.7%)	204.8 (-14.7%)
Mip-NeRF 360 (Speedy-Splat)	Baseline	27.11	0.799	0.263	1099.8	492.0
Mip-NeRF 360 (Speedy-Splat)	+ SkipGS (Ours)	27.08	0.799	0.264	1030.8 (-6.3%)	423.0 (-14.0%)

Consistent speedups and fidelity preservation hold across the other two benchmarks: - On Deep Blending, Vanilla 3DGS + SkipGS reduces \(T_{\text{post}}\) from 965.5s to 602.0s (-37.6%), providing a 21.2% end-to-end speedup while slightly improving PSNR (+0.09 dB, 29.79 to 29.88 dB). Taming 3DGS + SkipGS slashes \(T_{\text{post}}\) by 49.4% (620.0s to 314.0s). - On Tanks & Temples, Vanilla 3DGS + SkipGS lowers \(T_{\text{post}}\) by 36.8% (545.5s to 344.5s), speeding up end-to-end training by 20.2%. FastGS + SkipGS achieves an ultra-fast refinement time of 38.4s.

Ablation Study

On Mip-NeRF 360, the necessity of the backward budget control mechanism was evaluated by comparing full SkipGS against an unconstrained gating variant across GaussianSpa and Taming 3DGS:

Base Method	Setting	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	\(T_{\text{post}}\) (s) ↓	Note
GaussianSpa	Baseline	27.61	0.826	0.213	1485.0	Full unskipped backpropagation
GaussianSpa	+ SkipGS (Full)	27.60	0.825	0.215	1335.0	-150.0s refinement time with only -0.01 dB PSNR
GaussianSpa	+ SkipGS (w/o budget)	27.23	0.804	0.249	1026.0	Faster (-459s), but severe degradation (-0.38 dB)
Taming 3DGS	Baseline	27.94	0.822	0.207	757.0	Full unskipped backpropagation
Taming 3DGS	+ SkipGS (Full)	27.92	0.822	0.209	392.0	-365.0s (-48.2%) post-densification time, negligible drop
Taming 3DGS	+ SkipGS (w/o budget)	27.27	0.795	0.260	161.0	Over-aggressive skipping drops PSNR by 0.67 dB

Key Findings

• Budget control is essential for preventing quality collapse: Removing the budget mechanism allows gating to become overly aggressive, reducing \(T_{\text{post}}\) further at the cost of a disastrous 0.67 dB drop on Taming 3DGS and 0.38 dB on GaussianSpa. The auto-calibrated budget controller trades a minor fraction of speedup to guarantee stable convergence, preserving PSNR within 0.02 dB of the baselines.
• Strictly orthogonal and additive to primitive-reduction pipelines: Because SkipGS operates on the iteration axis rather than the spatial primitive axis, it delivers additive speedups across all types of base pipelines. Even when applied to FastGS—the fastest existing pipeline—it yields an additional 20.0% reduction in post-densification runtime.
• Zero primitive modification: Across all scenes, the final Gaussian primitive count remains strictly identical to the underlying baseline (e.g., 2.739M on Vanilla 3DGS), verifying that acceleration stems entirely from omitting non-informative backward passes without altering rendering-time throughput.

Highlights & Insights

• Phase-aware computation scheduling: Pinpoints the post-densification phase where backward passes account for over 60% of computation but generate flattened gradients, achieving substantial wall-clock savings without destabilizing structural densification.
• Minimalist statistical design: Tracks only a single floating-point scalar per viewpoint with EMA updates, incurring virtually zero memory overhead and no complex auxiliary neural network or policy training.
• Decoupled forward monitoring: Always evaluates forward rendering to update view baselines regardless of gating, preventing stale historical loss baselines from corrupting downstream skip tests.

Limitations & Future Work

• Static scene limitation: The authors acknowledge that SkipGS has been validated exclusively on static 3D scenes; in dynamic scenarios or 4D Gaussian Splatting, temporal motions and topological shifts induce higher loss variance, which may require motion-aware baseline estimation.
• Forward pass overhead remains: While skipping backward passes saves the majority of compute, the forward pass is still executed every step; combining SkipGS with coarse proxy caching or multi-view batching could yield further speedups.
• View-level gating granularity: Gating decisions are made per full image; extending the concept to tile-level or patch-level backward masking could unlock fine-grained computation savings for partially converged viewpoints.

Related Work & Insights

• vs Primitive Compaction (FastGS, LightGaussian, Speedy-Splat): Compaction methods reduce the number of active Gaussians per iteration, changing model capacity and requiring specialized rasterization tweaks; SkipGS schedules when to backpropagate, leaving primitive representations untouched and enabling immediate plug-and-play stacking.
• vs Growth Control (Taming 3DGS): Growth control regulates Gaussian proliferation during early densification; SkipGS optimizes the post-densification refinement regime, making the two approaches complementary across the two halves of training.
• vs Importance Sampling & Hard Example Mining: Unlike importance sampling methods that distort sampling probability distributions (introducing sampling variance or architectural changes), SkipGS retains uniform random camera sampling to preserve standard empirical risk minimization while dynamically gating the backward execution graph.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Pioneers orthogonal backward gating in 3DGS by exploiting post-densification convergence heterogeneity]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 3 standard benchmarks and 6 SOTA baselines with complete ablations and time breakdowns]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation backed by insightful gradient profiling and mathematically rigorous formulations]
• Value: ⭐⭐⭐⭐⭐ [Zero-cost plug-and-play integration for any existing 3DGS framework with immediate, reproducible speedups]