Matryoshka Gaussian Splatting

Conference: ECCV2026
Paper: ECCV Paper
Project: Matryoshka Gaussian Splatting
Area: 3D Vision
Keywords: Gaussian Splatting, continuous level of detail, nested representations, stochastic budget training, opacity ordering

TL;DR

MGS dynamically orders Gaussians by opacity and jointly trains a random-length prefix and the full set, enabling fine-grained budget truncation from one scene model with 28.20 dB full-budget PSNR and a quality–FPS AUC of 64.81 on MipNeRF 360, at roughly twice the backbone training cost.

Background & Motivation

3D Gaussian Splatting uses many explicit primitives for novel view synthesis, but mobile devices, mixed-reality headsets, and servers cannot afford the same rendering budget. Level of detail (LoD) is therefore more than offline compression: a running application may need to adjust the number of rendered Gaussians repeatedly. Conventionally trained Gaussians jointly explain the training images without an intrinsic importance order, so removing some can disrupt occlusion and accumulated color, producing holes and artifacts.

Discrete LoD approaches such as H3DGS and Octree-GS expose a limited number of levels and require auxiliary hierarchies. Models produced for separate compression targets need not contain nested primitive sets either. Continuous LoD better matches budgets that change with hardware and viewpoint, yet existing approaches can reduce full-capacity quality. The challenge is not merely rendering fewer Gaussians: small subsets must explain the main structure without preventing the full set from retaining reconstruction fidelity.

The paper transfers the independently usable prefix principle of Matryoshka representations from embedding dimensions to scene primitives. Core Idea: order Gaussians using their existing opacity parameters, then supervise a random-budget prefix and the complete set at every step, so early primitives support basic reconstruction and later ones add detail without a separate importance network or multiple LoD models.

Method

Overall Architecture

The input is a collection of posed training images, and the output is an ordered Gaussian set that supports prefix truncation. Each Gaussian retains its position, covariance, opacity, and view-dependent color parameters. MGS changes set organization and the training objective, not the renderer or Gaussian parameterization.

The implementation uses gsplat with 3DGS-MCMC for capacity control. Training determines the opacity-descending order, samples a keep ratio, and renders both the corresponding prefix and the full set from the same training camera. Parameters are updated jointly and then reordered. Deployment changes the prefix length without retraining for individual budgets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Posed images and fixed-capacity Gaussians"] --> Order["Opacity-Based Nested Ordering"]
    Order --> Train["Stochastic Joint Budget Training"]
    Train -->|Reorder after the update| Order
    Order -->|Truncate after training| Render["Standard rasterization and novel views"]

Key Designs

1. Opacity-Based Nested Ordering: retain major scene contributions in early prefixes

Gaussians lack a predefined ordered axis such as network channels. MGS uses each Gaussian's opacity as its importance score and constructs a descending permutation. Every budget selects the beginning of that permutation, ensuring that smaller sets are contained in larger ones instead of choosing a different independent subset each time.

The relationships in Eqs. (3)–(5) are:

\[ s(g_i)=\sigma_i,\qquad s(g_{\pi(1)})\ge\cdots\ge s(g_{\pi(N)}),\qquad \mathcal G_{\le k}=\{g_{\pi(1)},\ldots,g_{\pi(k)}\}. \]

This order determines which Gaussians are selected; it does not replace depth ordering inside standard front-to-back alpha compositing. Selected primitives still undergo ordinary camera-dependent rasterization. This distinction explains the unchanged rendering architecture and prevents confusing importance order with per-pixel compositing order.

Opacity also has a training advantage: it already receives per-pixel gradients through alpha compositing, avoiding a separate learnable importance scalar and its surrogate gradient. Position, volume, or spherical-harmonic energy is not automatically equivalent to reconstruction utility across budgets. The ablations show that good full-set quality alone does not establish reliable prefixes.

Sorting is not a one-time preprocessing operation. Every parameter update can change opacity, so the algorithm recomputes the descending permutation afterward, ensuring that the next sample uses the currently highest-ranked primitives. This is a discrete ordering operation that evolves with reconstruction training; the paper does not describe it as a differentiable network layer.

Nor is this simply pruning a frozen pretrained model: joint training reshapes the opacity distribution. Figure 6 reports a suppressed low-opacity tail and a 1.9-percentage-point increase in the high-opacity peak. The authors use this observation to explain why suffix truncation tends to discard contributions already weakened by training. It supports the mechanism but does not prove strictly monotonic quality for every prefix.

2. Stochastic Joint Budget Training: cover many budgets with two renders

Rendering all possible prefixes at every step would make training cost grow with the number of Gaussians. Instead, MGS samples one continuous keep ratio and rounds up to an integer primitive count. Reconstructing Eq. (6) from the accompanying text gives:

\[ r\sim\operatorname{Unif}(r_{\min},1),\qquad k=\lceil rN\rceil. \]

This covers the budget interval stochastically over training rather than training only a few predefined LoD levels. Here, continuous means fine-grained ratio control: the rendered primitive count remains an integer. Finite training also should not be described as visiting every possible prefix individually.

Training only random prefixes can favor low-budget reconstructions at the expense of full-set quality. MGS therefore also renders the complete set in the same iteration and supervises both outputs with the same ground-truth image. The full-set branch is neither a teacher nor a separate model; it uses the same Gaussian parameters at full capacity, and both losses update those shared parameters.

The full-set loss continuously constrains high-capacity quality, while the prefix loss forces early primitives to explain major visible structures independently. Their combination turns an ordering rule into a learned nested representation rather than a post-training removal heuristic. Each step needs 2 renders regardless of the number of candidate budgets, although sorting and the additional rendering still incur computational cost.

A Worked Example

Consider the main experiment's 5M-Gaussian capacity as an operational example, not an additional experimental result. If a sampled keep ratio is 10%, the current top 500,000 Gaussians by opacity render the prefix image, while all 5,000,000 Gaussians render the full image from the same training camera.

Both images are compared with the same ground truth. After the parameter update, reordering may move a Gaussian into or out of the next prefix. At deployment, 5%, 10%, and 100% budgets select 250,000, 500,000, and 5,000,000 Gaussians from the same model, without switching to separately trained parameters.

This example maps ratios to counts; it does not promise proportional FPS gains. Viewpoint, projected coverage, and rasterization overhead also affect throughput, which is why the paper evaluates both primitive count and actual FPS as efficiency axes.

Loss & Training

Each reconstruction loss follows the 3DGS weighted combination of L1 and D-SSIM. Denoting it by \(\ell\), the joint objective in Eq. (7) can be recovered from the surrounding prose as follows. Some cached equations have damaged typesetting, so the notation is normalized here rather than transcribed literally:

\[ \mathcal L_{\mathrm{MGS}}=\ell(\mathcal G_{\le k};I,c)+\gamma\ell(\mathcal G_{\le N};I,c),\qquad \gamma=1. \]

The prefix and full-set terms have equal default weights. Main experiments use \(N=5\mathrm M\), 50,000 training steps, and NVIDIA A100 GPUs on identical Ubuntu servers. MCMC supplies controlled capacity for budget comparisons; it is not an additional semantic module.

The main paper defines \(r_{\min}\), but the implementation paragraph in this cache does not specify its default value. It also does not explicitly give the numerical L1/D-SSIM mixing coefficient. The smallest evaluation ratio of 1% must not be treated as evidence of the training default. Exact reproduction requires checking the supplement or implementation.

Ablations use the single bicycle scene with 1M Gaussians and 50,000 steps, so their PSNR values cannot be directly merged with benchmark-wide main results. The weight sweep in Figure 8 gives the best AUC at a 6:4 prefix/full ratio, but the authors use 5:5 for simplicity and balanced performance. The default is not claimed to maximize AUC.

Key Experimental Results

Main Results

All four benchmarks reserve every 8th image for testing, including indices 0, 8, and 16. Continuous LoD methods are evaluated at 12 ratios: 100%, 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, 5%, and 1%. Discrete methods use their recommended operating points.

The following selected results come from Table 1. PSNR/SSIM/LPIPS describe each method at its highest budget, whereas AUC summarizes a cross-budget curve; these are different measurements.

Benchmark	Method	PSNR↑	SSIM↑	LPIPS↓	AUCfps↑	AUCsplats↑
MipNeRF 360	3DGS-MCMC	28.40	0.843	0.133	—	—
MipNeRF 360	Octree-GS	27.62	0.813	0.221	21.61	68.43
MipNeRF 360	CLoD-3DGS	27.44	0.814	0.215	46.38	56.56
MipNeRF 360	MGS	28.20	0.841	0.130	64.81	77.79
Tanks & Temples	MGS	24.56	0.874	0.086	70.16	77.54
Deep Blending	Octree-GS	30.35	0.910	0.252	29.54	83.21
Deep Blending	MGS	28.41	0.902	0.176	82.63	83.22
BungeeNeRF	Octree-GS	28.23	0.922	0.088	19.69	73.70
BungeeNeRF	MGS	27.13	0.906	0.088	62.99	79.93

For AUC, the paper first linearly normalizes and clamps PSNR, SSIM, and LPIPS to [0, 1] using fixed ranges (14, 32), (0.35, 0.92), and (0.06, 0.60), respectively. The resulting tilde-marked metrics define the composite quality in Eq. (9):

\[ \bar Q=\frac{\tilde p+\tilde s+(1-\tilde\ell)}{3}. \]

AUCfps integrates a monotone quality–FPS envelope with operating points extended toward lower throughput over [0, 500] FPS. AUCsplats extends points toward larger primitive counts, connects the origin to the lowest-budget point, and integrates over [0, 5M]. Both scores are normalized and multiplied by 100. Thus, 64.81 is an AUC score, not an FPS or PSNR measurement.

Ablation Study

The following rows are selected from Table 3, all using bicycle and 1M Gaussians. Ordering and budget-objective ablations have separate default rows: the reported 25.47/68.62 and 25.43/66.87 should remain distinct rather than being merged into one supposedly universal default.

Ablation Group	Configuration	Full-Budget PSNR↑	LPIPS↓	AUCfps↑	AUCsplats↑
Ordering	Opacity descending	25.47	0.174	68.62	61.05
Ordering	Opacity ascending	24.91	0.219	30.53	33.48
Ordering	SH energy descending	25.17	0.187	59.39	47.12
Ordering	Fixed append	25.51	0.175	46.41	43.09
Budget objective	Prefix + full	25.43	0.175	66.87	60.96
Budget objective	Prefix only	24.97	0.253	56.58	57.09
Budget objective	Fixed MRL nesting	25.37	0.183	66.57	60.35

Key Findings

• Similar full-set quality does not establish low-budget usability. Fixed append reaches 25.51 PSNR versus 25.47 for opacity descending, but its AUCfps is only 46.41 versus 68.62. The analysis associated with Figure 7 reports 22.2 dB at 493 FPS for opacity descending at 10% budget, versus 17.6 dB for SH energy descending.
• Removing full-set supervision reduces full-budget PSNR from 25.43 to 24.97 and increases LPIPS from 0.175 to 0.253. This directly supports retaining the full-set branch instead of relying only on random-prefix training.
• Table 2 reports training time increasing from 2592 s to 5325 s on MipNeRF 360, a 2.05-fold cost. The other benchmarks report factors of 2.12, 2.37, and 2.06. The two renders are a training cost, not a requirement to render twice at inference.
• Preserving full-capacity quality means remaining close to the backbone, not avoiding every decrease: MGS is 0.20 dB below MCMC on both MipNeRF 360 and Tanks & Temples. Its lowest BungeeNeRF LPIPS of 0.088 is tied with Octree-GS.

Highlights & Insights

• Training, not the sorting function alone, makes the ordered representation useful. MGS adapts the model to missing suffixes before opacity-based truncation, giving it a stronger mechanism than simply sorting and truncating an arbitrary pretrained Gaussian model.
• A persistent full-set anchor reduces continuous-budget supervision to two renders per step. Other explicit primitive representations could reuse the idea, provided that an importance scalar works coherently with task gradients.
• Combining full-budget quality with cross-budget envelopes exposes methods that excel at only one operating point. Fixed append versus opacity descending in Table 3 illustrates why single-point PSNR is insufficient for LoD evaluation.

Limitations & Future Work

• The authors explicitly identify view-agnostic ordering as a current limitation and view/spatial adaptation as future directions. Globally early Gaussians need not be most useful to the current camera, and adaptive scheduling and device-aware rendering are not yet complete solutions.
• Full-set quality does not always win. In the DrJohnson failure case of Figure 5, CLoD-3DGS reaches 29.1 dB at 100% budget versus 27.7 dB for MGS, although MGS degrades more gracefully at lower budgets.
• Most ablations focus on bicycle, limiting evidence that ordering and weight choices are equally stable across scenes. A100 experiments also do not directly establish latency, power consumption, or temporal stability on phones and headsets.
• Evaluation caveat from this note: methods use different backbones and their respective highest budgets, while AUC depends on fixed normalization ranges and integration limits. The results are not a strictly equal-capacity proof of architectural superiority.
• The paper does not prove strictly monotonic quality for every integer prefix or useful reconstructions at arbitrarily tiny budgets. Further work could add multi-seed evaluation across more scenes, device measurements, and temporal consistency tests during continuous budget changes.

Related Work & Insights

• vs 3DGS-MCMC: the backbone supplies fixed-capacity training, while MGS adds the joint prefix/full objective. This control isolates the cost of gaining LoD more directly than peak-PSNR comparisons across different backbones.
• vs Octree-GS / H3DGS: these methods expose discrete levels through octree anchors or hierarchical thresholds; MGS uses prefixes of a single ordered set. It offers finer budget granularity without winning peak PSNR on every scene.
• vs CLoD-GS / CLoD-3DGS: the paper describes them as distance-dependent opacity decay and learned importance ordering, respectively. MGS emphasizes the combination of existing opacity parameters, dynamic reordering, and full-set supervision, rather than being the first method to vary Gaussian count.
• vs Matryoshka Representation Learning: both share nested capacity, but Gaussians initially lack a fixed order and interact through occlusion during rendering. Stochastic budgets provide finer training coverage than fixed prefix sizes.

Rating

• Novelty: 4/5. Extends nested representations to reorderable Gaussian primitives with a simple, targeted objective.
• Experimental Thoroughness: 4/5. Four benchmarks, six LoD comparisons, and ordering/objective/weight ablations, with limited device and cross-scene stability evidence.
• Writing Quality: 4/5. Clear mechanism and failure case, but some reproduction settings are absent from the main text and absolute quality claims require reading the tables carefully.
• Value: 4/5. Useful for single-model, multi-budget novel view rendering, subject to additional training cost.