Drop-In Perceptual Optimization for 3D Gaussian Splatting¶

Conference: ECCV 2026
Paper: ECCV page / project page
Area: 3D Vision
Keywords: 3D Gaussian Splatting, perceptual optimization, Wasserstein Distortion, human preference study, rate–distortion compression

TL;DR¶

Treating the 2D distortion loss used in 3DGS training as the single design variable, this paper compares the original L1+SSIM loss, a composite L1+L2+MS-SSIM+LPIPS loss, and Wasserstein Distortion (WD), and picks the winner with a large-scale subjective study of 39,320 pairwise ratings: a lightly regularized WD variant, WD-R, replaces the loss and nothing else, yet is preferred by raters more than 2.3× over the original loss and 1.5× over Perceptual-GS, and it transfers unchanged to Mip-Splatting, Scaffold-GS, and 3DGS compression (≈50% bitrate savings).

Background & Motivation¶

3DGS renderings are ultimately consumed by human viewers, yet most 3DGS methods are trained with losses that were assembled ad hoc — the L1+SSIM combination introduced by Kerbl et al. — and those pixel-level objectives tend to produce over-smoothed, blurry textures. Perceptual metrics such as LPIPS are routinely used for evaluation, but whether they can serve as optimization objectives has never been rigorously investigated, and none of the various perceptual optimization techniques has been validated against human preference at all. Ignoring how the content will be perceived wastes representational capacity: every Gaussian and every bit spent on imperceptible detail buys no perceptual benefit, which becomes critical once a 3DGS representation has to be compressed for storage or transmission.

Existing remedies attack the problem only indirectly or partially. Perceptual-GS uses Sobel-based edge detection to steer densification, Pixel-GS rewrites pixel-aware gradient accumulation and pruning, FreGS applies frequency regularization, and EGGS uses edge guidance. All of them modify the model, the optimization procedure, and the loss at the same time, coupling human-vision modeling to the 3DGS algorithm itself — which makes the source of the gains hard to isolate and the solution hard to transfer to arbitrary scenes.

The real tension is that in 3DGS the loss is not an ordinary objective: because adaptive densification is triggered by gradient magnitude, the scale of the loss influences how many Gaussians are created, so a "stronger" perceptual loss may be quietly buying quality with more splats and larger models — comparisons across losses are only meaningful once the representation budget is aligned. On top of that, perceptual metrics like LPIPS are pointwise deep-feature distances that do not match how humans perceive texture, so metrics alone cannot declare a winner. The angle this paper takes is therefore a clean decoupling: freeze initialization, densification, and pruning, replace only the distortion term \(D(\cdot)\) in the loss, and let a large-scale human preference study — the first of its kind for 3DGS — act as the judge. Core idea: outsource the modeling of human vision entirely to a 2D distortion term that compares local statistics (local mean and standard deviation) in VGG feature space rather than pointwise pixel differences, and add a light pixel-level regularizer (WD-R), yielding perceptual gains that human raters confirm without changing any 3DGS algorithm or inflating the splat count.

Method¶

Overall Architecture¶

The method itself is minimal: no new module, no architectural change — it swaps the function that computes the distortion between the rendered image and the ground truth inside the 3DGS training loop. Standard 3DGS initializes a set of 3D Gaussian primitives from SfM points, each parameterized by center, covariance, color, and opacity; a differentiable rasterizer projects them to 2D, alpha-blends them in depth order, and produces a rendered image \(\hat{x}\), which is compared against the ground-truth image \(x\). That distortion produces gradients along two paths: one updates the Gaussian parameters, the other drives adaptive densification (adding primitives) and pruning of low-opacity ones according to gradient magnitude — the objective therefore governs both how parameters move and what the representation grows into.

The paper's change is exactly this one place: the distortion term \(D(x,\hat{x})\) is replaced by the composite loss, WD, or WD-R, while everything else (initialization, rasterizer, densification thresholds, pruning rules) stays untouched, with no 3D-specific constraints or heuristics introduced. Two task settings are studied: the representation task minimizes \(\gamma\,D(x,\hat{x})\), and the variable-rate compression task adds the rate term \(\lambda R_\theta(G)\) induced by entropy coding, trading perceptual quality against storage cost. Since the whole pipeline is differentiable, "drop-in" means, in practice, swapping the loss computation inside the training loop, plus a short warm-up phase that keeps the original loss (see Loss & Training below).

Key Designs¶

1. Replacing pointwise distortion with WD: comparing local statistics in VGG feature space, not pixels

L1, SSIM, and LPIPS share one weakness — they compare images pointwise. The paper makes this concrete with a 1D sketch: two textures differing only by a 180° phase shift have a huge pointwise difference yet look nearly identical to a human. WD changes the comparison itself: for each VGG feature map and each spatial location it first computes a local mean \(\mu\) and a local standard deviation \(\nu\) over a pooling kernel of size \(\sigma\), then compares those statistics and aggregates the differences over feature maps and locations. Per the paper's prose, the form is

\[d_{\text{WD}} = \mathrm{RMSE}\Big(\{\mu_{l,p}-\hat{\mu}_{l,p}\}\cup\{\nu_{l,p}-\hat{\nu}_{l,p}\}\Big)\]

where hatted quantities are the local statistics computed on the reconstructed image and \(\mu,\nu\) are the local first and second moments. ⚠️ Equation (4) is mangled in the cached text; this expression is reconstructed from the prose — refer to the original paper for the exact form.

The justification comes from models of the human visual system: peripheral vision encodes images through summary statistics over local pooling regions rather than precise pointwise pixel values (metamers of the ventral stream; Rosenholtz). Consequently, textures that differ substantially pointwise are considered perceptually similar once their local statistics match — which is precisely what frees texture resampling, something humans are insensitive to, from the optimization objective. The pooling kernel \(\sigma\) is the dial of this metric: a larger \(\sigma\) aggregates over broader regions and is more permissive to texture resampling, while \(\sigma \to 0\) converges to a pointwise distance. The paper fixes \(\sigma=4\) and explicitly describes it as a strong setting for its evaluation setup rather than a universal optimum (the optimal \(\sigma\) may depend on input resolution and viewing distance). Since local means and variances are differentiable with respect to the input, gradients flow back through the pooling to the Gaussian parameters, so the optimizer is directly rewarded for matching texture statistics rather than exact pixel values; the geometric consequence is that Gaussians become more anisotropic and needle-like, using elongated shapes to conform to local structure and represent high-frequency detail and fine texture better.

2. WD-R: a touch of the original pixel loss to suppress WD's "web-like" artifacts

WD on its own is not always stable as an objective. The authors find it fails in two situations: in regions of 3D space with few training samples, and when the splat budget is tightly constrained. In those cases WD can satisfy the local statistics with a few highly elongated Gaussians (it only has to match means and standard deviations in a deep feature space), producing web-like filaments that crudely — but economically — imitate texture. Pixel-level losses never do this, but instead smooth texture away. Enlarging the global scale factor (i.e. granting more splats) mitigates the artifacts too, but pushes the count from 4.5M to 7.2M.

WD-R therefore augments WD with a lightly weighted version of the original pixel-level loss:

\[\mathcal{L}_{\text{WD}} = \gamma\, d_{\text{WD}}, \qquad \mathcal{L}_{\text{WD-R}} = \gamma\,(d_{\text{WD}} + \beta\,\mathcal{L}_{\text{orig}})\]

where \(\mathcal{L}_{\text{orig}}\) is the original L1+SSIM loss, \(\gamma\) is the dataset-specific global scale (design 3), and \(\beta\) is chosen so that the pixel term acts only as a mild pixel-fidelity regularizer while WD remains the dominant perceptual term. The result is that artifacts are suppressed at essentially no extra cost in splats (4.5M → 4.4M), with better perceptual metrics and better human preference at the same time. Notably, the discussion flags a counter-intuitive observation: WD alone is perfectly stable as a distortion loss for 2D images in learned image compression, but this is not entirely the case for 3D scene optimization, and the root cause remains an open question — the paper's answer is an empirical regularizer.

3. Using the global scale γ to decouple loss magnitude from the splat budget

Because densification is triggered by gradient magnitude, the overall scale of the loss indirectly determines how many Gaussians appear — which means a naive comparison of two losses conflates "which loss is better" with "how much representation capacity was granted." The paper's handling is to adjust only the global scale \(\gamma\) (per dataset) in the equation while freezing every other hyper-parameter, so that the resulting Gaussian counts of different methods align, and all comparisons are made at similar or lower budgets (fewer splats, or smaller model size). Though it looks like mere experimental protocol, this is the premise that lets the method be called drop-in: it ensures WD-R's gains come from what the loss looks at, not from more capacity being handed out. The results bear this out — WD-R leads on perceptual metrics while being more compact, e.g. cutting the Gaussian count on BungeeNeRF from 6.92M (original loss) to 4.89M.

Loss & Training¶

The representation task minimizes \(\gamma D(x,\hat{x})\); the compression task minimizes \(\min_{G,\theta} D(x,\hat{x}(G)) + \lambda R_\theta(G)\), where the rate term takes the standard neural-compression form \(R_\theta(G)=\mathbb{E}[-\log_2 p_{\hat{y}}(\hat{y})]\) — the Shannon cross-entropy between the quantized latent encoding \(\hat{y}\) of the Gaussians and a learned parametric entropy model \(p_{\hat{y}}\) — and \(\lambda\) controls the quality/storage trade-off. Storage size is not determined by Gaussian count alone but also by parameter redundancy and the probabilistic model used for entropy coding. Compression experiments sweep \(\lambda \in [3^{-2}, 3^{-5}]\).

The composite loss is \(\mathcal{L}_{\text{comp}} = \omega_1\mathcal{L}_{L1} + \omega_2\mathcal{L}_{L2} + \omega_3\mathcal{L}_{\text{MS-SSIM}} + \omega_4\mathcal{L}_{\text{LPIPS}}\), with weights selected by ablation and fixed throughout at \(\omega_1=0.05,\ \omega_2=0.30,\ \omega_3=0.60,\ \omega_4=0.10\). One LPIPS detail is worth recording: training uses LPIPS-AlexNet while evaluation uses LPIPS-VGG, to avoid overfitting to the training metric and to stay consistent with standard 3DGS evaluation practice.

All models are trained for 30k iterations using the published configurations of the respective baselines. WD-based objectives do not take over from the start: the model is first optimized with the standard 3DGS loss for 3k–5k iterations before the perceptual objective is introduced, mirroring warm-up strategies in perceptual and adversarial training — which is why "where the loss is inserted" is a detail this method has to spell out. On cost, the WD implementation adopted is unoptimized and incurs roughly a 4.5× per-iteration training overhead over original 3DGS; caching the ground-truth VGG features and pruning zero-weight VGG pyramid levels reduces WD's per-iteration time by ≈48% with the loss and gradients remaining bit-exact, bringing the relative overhead down to ≈2.8×. This is arguably the most practical cost item when deploying the method.

Key Experimental Results¶

Main Results¶

Perceptual metrics for the studied distortion losses on 21 scenes across four datasets (Mip-NeRF 360 with 9 scenes, split into 4 indoor and 5 outdoor; Deep Blending 2; Tanks & Temples 2; BungeeNeRF 8), following Perceptual-GS's indoor/outdoor split (excerpt of the paper's Table 1):

Dataset	Method	#G	LPIPS↓	DISTS↓	FID↓
Mip-NeRF 360 (Indoors)	Original loss (L1+SSIM)	1.42M	0.188	0.158	80.70
Mip-NeRF 360 (Indoors)	Pixel-GS	2.49M	0.177	0.147	73.36
Mip-NeRF 360 (Indoors)	Perceptual-GS	1.58M	0.170	0.142	69.86
Mip-NeRF 360 (Indoors)	Composite	1.99M	0.171	0.143	82.17
Mip-NeRF 360 (Indoors)	WD	1.46M	0.152	0.117	65.59
Mip-NeRF 360 (Indoors)	WD-R	1.49M	0.147	0.114	67.80
Mip-NeRF 360 (Outdoors)	Original loss	4.52M	0.244	0.218	104.88
Mip-NeRF 360 (Outdoors)	Pixel-GS	7.40M	0.206	0.186	65.49
Mip-NeRF 360 (Outdoors)	Perceptual-GS	3.55M	0.206	0.188	58.97
Mip-NeRF 360 (Outdoors)	Composite	6.50M	0.216	0.199	59.09
Mip-NeRF 360 (Outdoors)	WD	3.54M	0.228	0.178	65.69
Mip-NeRF 360 (Outdoors)	WD-R	3.47M	0.206	0.168	59.25
BungeeNeRF (Outdoors)	Original loss	6.92M	0.098	0.106	62.23
BungeeNeRF (Outdoors)	Pixel-GS	OOM (Pompidou scene)	—	—	—
BungeeNeRF (Outdoors)	Perceptual-GS	4.97M	0.095	0.103	58.23
BungeeNeRF (Outdoors)	Composite	11.30M	0.197	0.200	101.66
BungeeNeRF (Outdoors)	WD	4.67M	0.116	0.100	50.68
BungeeNeRF (Outdoors)	WD-R	4.89M	0.092	0.087	46.21

⚠️ Numbers are taken from Table 1 as it appears in the cached text; that table suffers from column bleed in the cache, and a few cells (e.g. CMMD in the outdoor block) cannot be recovered reliably, so only LPIPS/DISTS/FID are listed here. Refer to the original paper for exact values.

Human preference study (Bradley–Terry model with Bayesian Elo aggregation; blind A/B of the same 704×704 random crop against the ground-truth reference):

Comparison	Elo difference	Preference ratio	Scale
WD-R vs original loss	>150	>2.3×	main study: 320 participants, 30,720 pairwise trials (indoor + outdoor)
WD-R vs Perceptual-GS	72	>1.5×	same as above
WD-R vs Mip-Splatting	105.7	1.8×	86 participants, 4,880 votes
WD-R vs Scaffold-GS	223.2	3.6×	93 participants, 3,720 votes

Across all studies, 428 participants completed 39,320 pairwise ratings. In the compression setting, WD-based objectives also give the best LPIPS/DISTS/FID across the entire rate–distortion curve, translating to ≈50% bitrate savings at comparable perceptual quality.

Ablation Study¶

Comparison of the loss formulations themselves (BungeeNeRF outdoor 8 scenes, with each method's resulting splat count):

Config	#G	LPIPS↓	DISTS↓	Note
Original loss L1+SSIM	6.92M	0.098	0.106	baseline; over-smoothed textures
Composite L1+L2+MS-SSIM+LPIPS	11.30M	0.197	0.200	stacking more metrics is uniformly worse and inflates splats by 63%; excluded from the human study for this reason
WD (no regularization)	4.67M	0.116	0.100	most splat-efficient and better DISTS than the original loss, but LPIPS is worse and web-like artifacts appear
WD-R (ours)	4.89M	0.092	0.087	best on all three perceptual metrics, with ≈29% fewer splats than the original loss

Generalization across frameworks (only the training objective is modified; the framework is left unchanged):

Framework	Dataset	Metric	Original	+ WD	+ WD-R
Mip-Splatting	Mip-NeRF 360 Indoors	#G / LPIPS↓	1.81M / 0.152	1.55M / 0.134	1.69M / 0.123
Mip-Splatting	Mip-NeRF 360 Outdoors	#G / LPIPS↓	5.71M / 0.193	5.68M / 0.202	5.37M / 0.181
Scaffold-GS	Mip-NeRF 360 Indoors	model size / LPIPS↓	99.5 MB / 0.166	98.4 MB / 0.143	93.9 MB / 0.140
Scaffold-GS	Mip-NeRF 360 Outdoors	model size / LPIPS↓	220.7 MB / 0.273	215.8 MB / 0.245	223.5 MB / 0.233

Key Findings¶

The gains come from "what the loss looks at," not from "how much capacity was given." WD-R achieves better perceptual metrics with fewer splats on several datasets (BungeeNeRF 6.92M → 4.89M), which is exactly what the budget-alignment protocol of design 3 exists to establish; on Scaffold-GS the model size is essentially unchanged (even slightly smaller) while LPIPS drops from 0.166 to 0.140.
The three perceptual metrics do not always agree — which is precisely why the paper runs a human study. On BungeeNeRF, WD's LPIPS (0.116) is worse than the original loss (0.098), yet the same configuration is better on DISTS (0.100 vs 0.106), FID (50.68 vs 62.23), and CMMD. A single metric cannot adjudicate; only after adding the \(\beta\) regularizer does WD-R overtake on every metric.
The PSNR/SSIM trade-off cannot be verified from the main text. Only perceptual metrics are reported in the body; PSNR and SSIM are deferred to Appendix E.1, so whether perceptual gains come at a PSNR cost cannot be read off the cached text (⚠️ consult the appendix). What is certain: the human study was blind and always presented the ground-truth reference alongside the two candidates, so the gain is real at least along the human-preference axis — and the paper does not claim PSNR improvements.
Naively stacking metrics fails. Putting LPIPS directly into the optimization objective makes the composite loss worse than the original loss on most datasets and inflates the splat count badly (BungeeNeRF 11.30M), showing that the solution is not "add a perceptual metric" but "redefine the distortion."
Perceptual optimization changes the geometry of the Gaussians. WD-based objectives shift Gaussians toward stronger anisotropy: on the Barcelona scene the median effective rank (erank) of the covariance matrices drops from 1.55 to 1.12 (erank near 1 means needle-like, near 3 means isotropic; defined as \(\text{erank}(G_k)=\exp(-\sum_{i=1}^{3}q_i\log q_i)\) with \(q_i=s_i^2/\sum_j s_j^2\) over the singular values \(s_i\) of the covariance matrix). Splat density heatmaps show WD-R concentrating capacity on the texture-rich cathedral while the original loss spreads splats more uniformly across the view.
The failure boundary is clearly delimited. WD used alone produces web-like artifacts in regions with few training views and a tightly constrained splat budget (Bicycle scene); mitigating this by raising \(\gamma\) pushes the count from 4.5M to 7.2M, whereas WD-R suppresses it at 4.4M. Pixel-GS runs out of memory on the Pompidou scene of BungeeNeRF.

Highlights & Insights¶

Human-vision modeling is fully separated from the 3DGS algorithm, leaving \(D(\cdot)\) as the only interface. Prior work (Perceptual-GS's Sobel edges, Pixel-GS's pixel-aware gradients, FreGS's frequency regularization) couples perceptual modeling with densification strategy, making attribution hard and transfer harder; compressing it into a loss swap lets the method be stacked orthogonally onto Mip-Splatting, Scaffold-GS, and Comp-GS. The research taste of "collapse the variable to one place" is itself worth borrowing.
Defining distortion over local statistics rather than pointwise values is what frees texture resampling from the loss. Textures 180° out of phase are worlds apart in pixel space and nearly identical in the local mean/variance space — this single observation explains both why WD recovers detail and why it cheats with web-like filaments when the budget is tight (matching statistics is not the same as reconstructing structure). Two sides of one coin.
\(\sigma\) is an interpretable dial: \(\sigma \to 0\) degenerates to pointwise distance while larger \(\sigma\) is more permissive. It gives a continuous "perceptual budget" axis with more physical meaning than hand-tuned loss weights.
Using a human preference study (Bradley–Terry / Bayesian Elo) as the final arbiter, and reading the Elo difference as a preference ratio (a difference above 150 corresponds to 2.3×), is a first for 3DGS; for any work claiming "better perception," this is a reproducible and affordable validation paradigm.
Directly transferable: any 3D representation learning supervised by 2D renderings (NeRF, 2DGS, avatar/human reconstruction, video reconstruction) can swap \(D\) for WD-R. On the compression side it is even more direct — the rate term is unchanged and perceptual quality improves, which converts into bitrate savings at equal quality (≈50% measured here) and can be folded into existing neural-compression-style 3DGS pipelines as is.
A concrete engineering trick: caching the ground-truth VGG features and pruning zero-weight VGG pyramid levels saves ≈48% of per-iteration time with a bit-exact loss and gradients. Any work using a frozen feature network in its loss can copy this.

Limitations & Future Work¶

The authors acknowledge that WD alone produces web-like artifacts as a 3D scene optimization objective, and that the root cause and a principled fix remain open questions; WD-R is an empirical regularizer whose \(\beta\) must be tuned.
\(\sigma=4\) is a "strong setting for the evaluation setup, not a universal optimum"; the best value may depend on input resolution and viewing distance. Spatially varying \(\sigma\) (Appendix C.2) occasionally improves the reconstruction of structured textures (e.g. English text) but yields overall metrics similar to fixed \(\sigma=4\) while adding complexity.
Cost remains a hard constraint: ≈2.8× the training time of original 3DGS, and a VGG feature dependency. The authors also note that adversarial losses would be even more expensive, and that incorporating splat count explicitly into a fully end-to-end optimized objective is left for future work. The main text provides no complete training-time/memory comparison table.
The strength of the human evidence is uneven: the main study has 320 participants and 30,720 comparisons, but the two generalization claims rest on smaller sub-studies of 86 and 93 participants (4,880 / 3,720 votes), and the "1.8×"/"3.6×" conversions rely entirely on Bradley–Terry/Elo assumptions without reporting the confidence intervals numerically.
Additional observations of my own: the comparison tables tie WD-R's gains to "similar or lower splat budgets," but budget alignment is achieved by tuning \(\gamma\) per dataset by hand, which introduces a per-dataset search cost the paper never quantifies; and pushing all PSNR/SSIM numbers to the appendix weakens the verifiability of the question readers care about most — whether perceptual gains are bought with fidelity.
Possible improvements: give the WD artifacts a structural explanation (e.g. restrict local-statistic matching to regions covered by enough training views) to reduce reliance on the regularizer; bind \(\sigma\) to resolution and view density as an adaptive policy instead of an engineering patch like spatially varying \(\sigma\); and write splat count / model size explicitly into the objective so the perception–capacity trade-off can be optimized end to end rather than aligned externally through \(\gamma\).

vs original 3DGS (Kerbl et al.): They defined the de facto standard L1+SSIM loss; this paper swaps only the loss inside the same framework with the same densification strategy, making the comparison maximally clean. The result — >2.3× human preference with fewer splats (BungeeNeRF 4.89M vs 6.92M) — indicates the original loss really is the perceptual bottleneck.
vs Pixel-GS / Perceptual-GS: Both improve perceptual sharpness by changing density control (pixel-aware gradient accumulation / Sobel-based edge-guided densification) and explicitly constrain Gaussian growth. This paper changes only the loss and wins on perceptual metrics and human preference at equal or lower budgets; Perceptual-GS also tends to overemphasize linear structures on the Barcelona scene, suggesting the limits of an edge heuristic prior. The two directions are orthogonal and could in principle be combined.
vs FreGS / EGGS / Analytic-Splatting / Mip-Splatting: Frequency regularization, edge guidance, and analytic-integration/multi-scale filtering belong to densification heuristics and anti-aliasing architecture respectively. Using Mip-Splatting as a testbed shows the perceptual loss swap is complementary to such architectural improvements (+WD-R reaches LPIPS 0.123 indoors and 0.181 outdoors).
vs Scaffold-GS: Structured anchor-based representations measure compactness by model size rather than splat count; changing only the training objective drops LPIPS from 0.166/0.273 to 0.140/0.233 with model size essentially unchanged or smaller, showing perceptual optimization is orthogonal to representation structure as well.
vs Comp-GS / the neural compression line: These bring rate–distortion optimization into 3DGS; this paper shows that swapping the perceptual loss while leaving the rate term alone buys ≈50% bitrate savings. Conversely, WD is stable in image-compression literature yet needs regularization for 3D scene optimization — a discrepancy that is itself an open thread worth pursuing.

Rating¶

Novelty: ⭐⭐⭐⭐ The method itself ports an existing WD metric into 3DGS and adds a pixel regularizer (WD-R), a modest technical increment; but framing perceptual loss as the single variable in a systematic study, plus the first large-scale human preference study in 3DGS, are genuine contributions.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 21 scenes from 4 datasets × 3 framework families (standard / anti-aliased / structured) plus compression rate–distortion curves, 39,320 human ratings, and mechanistic analyses via erank and splat-density heatmaps — including an honestly reported failure case (Bicycle web artifacts).
Writing Quality: ⭐⭐⭐⭐ Clear structure and restrained claims (it explicitly states \(\sigma=4\) is not universally optimal, the artifact root cause is unknown, and PSNR lives in the appendix), and the discussion does not dodge the counter-intuitive findings; points off because key fidelity metrics are pushed to the appendix and the cost of budget-alignment search is never quantified.
Value: ⭐⭐⭐⭐⭐ An almost zero-integration-cost, drop-in, human-preference-validated loss replacement that works directly on existing 3DGS pipelines; the ≈50% bitrate savings on the compression side carries clear practical value as well.