3D Gaussian Splatting Compression with Object Scalability

Conference: ECCV2026
Official Paper: 4180
Paper: Conference PDF
Code: https://github.com/RuixiangXue/ScalableGSC
Area: 3D Vision
Keywords: 3D Gaussian compression, object scalability, progressive distillation, cross-layer context, region of interest

The code URL is supplied by the paper, which describes its release as forthcoming. Its public availability was not checked online for this note.

TL;DR

RecastGS reorganizes pretrained 3DGS into a hierarchy with object-specific quality choices, and LayeredCGS compresses it for progressive decoding; on kitchen near 2.3 MB, ROI PSNR rises from 19.65 to 21.81 dB, at the cost of lower background and full-image quality.

Background & Motivation

3D Gaussian Splatting represents a scene using many explicit primitives, each carrying position, rotation, scale, opacity, and spherical-harmonic color. This supports real-time rendering but can produce scene models exceeding 500 MB. Existing compression methods generally optimize the scene as a whole, either learning a compact representation during reconstruction or pruning, quantizing, and entropy-coding an existing model. FCGS makes compression a generalizable feed-forward operation and reduces per-scene optimization costs, but still treats the scene holistically.

Interactive viewing has a different objective. A user may care about one object in a kitchen while needing the background only for spatial context. If foreground and background must share a quality level, bandwidth cannot be directed specifically toward that object. Progressive methods such as GoDe and PCGS already introduce a hierarchy, but global quality scalability does not provide object-level allocation. Simply dividing an unordered set of Gaussians into layers also does not ensure that every cumulative subset renders well by itself.

The paper therefore separates representation reorganization from coding: first make subsets of different capacities useful, then overlay object membership on the quality hierarchy, and finally use decoded lower layers as context for higher-layer coding. Core idea: decouple how many layers are retained from which object receives those layers, allowing a coarse background to support a detailed foreground while the scene is transmitted and refined progressively.

Method

Overall Architecture

The input is a pretrained 3DGS model, not a new scene reconstruction from images. RecastGS first applies overcomplete layer search to establish an importance hierarchy; optional progressive distillation improves its cumulative submodels. A parallel prompted object-extraction branch assigns Gaussians to foreground or background, and these assignments are combined with the hierarchy for region-adaptive quality allocation. LayeredCGS then compresses positions and attributes into layer-organized bitstreams, letting the receiver decode a base layer before adding enhancement layers.

A quality level means the cumulative model containing the first several layers, not an isolated layer. Independent bitstream segments also should not be interpreted as arbitrary enhancement layers that can be decoded without a base: later entropy models condition on previously decoded layers. Progressive reception must respect that dependency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pretrained 3DGS"] --> B["Overcomplete Layer Search"]
    B --> C["Progressive Distillation<br/>Optional"]
    A --> D["Prompted Object Extraction<br/>and Region Recomposition"]
    C --> D
    D --> E["Cross-Layer Conditional Coding"]
    E --> F["Layered Bitstream<br/>Preview and Refinement"]

Key Designs

1. Overcomplete Layer Search: select at fine granularity, then merge into delivery layers

The base layer must support the entire scene with relatively few Gaussians, while higher layers add detail. The paper builds on GoDe's gradient-based importance hierarchy: within the remaining set, it computes parameter gradients of the rendering loss over training views and accumulates their norms as Gaussian importance. Working from high layers toward low layers, it removes less important primitives into higher layers, preserving important ones in the lower layers. Cumulative Gaussian counts grow exponentially, providing useful size control at low quality levels.

Direct selection with the final layer count can make importance updates too coarse, whereas retaining many layers increases subsequent optimization costs. RecastGS searches 16 fine layers and merges each adjacent pair into one of 8 final layers, with the base-layer count parameter set to 100,000 Gaussians. Selection benefits from the finer partition without forcing distillation and delivery to use all fine layers. This improves an existing gradient hierarchy; it does not introduce a new semantic object-importance score.

2. Progressive Distillation: shrink capacity from a submodel already close to the teacher

After pruning, surviving Gaussians still carry parameters optimized for the complete scene, not necessarily for rendering on their own. GoDe's finetuning randomly samples a quality level and supervises its cumulative subset using ground-truth images. Small subsets can struggle to fit all high-frequency details, and random sampling may distribute optimization unevenly across levels. RecastGS instead uses the fixed, complete pretrained model as a teacher and supervises the student with rendered teacher images, avoiding the need to access all original training images for this finetuning step.

The schedule runs from level 7 down to level 1, rather than growing from the smallest model. Each student inherits the optimized parameters of the preceding, larger submodel, removes higher-layer Gaussians, and continues fitting the full teacher with less capacity. The full level-8 model corresponds to the original pretrained model. The essential objective in Eq. (4) is squared image error:

\[ \mathcal{L}_{\mathrm{distill}}=\left\|I_T-I_{S_i}\right\|_2^2. \]

Teacher images continue to come from the full model throughout. What passes between stages is the student's initialization, not a replacement teacher. Each stage therefore compensates for a nearby reduction in capacity rather than asking the smallest model to bridge the whole gap immediately. Distillation uses 30,000 iterations and is optional per-scene optimization. The term feed-forward compression accurately describes LayeredCGS, not the entire pipeline with distillation enabled.

3. Prompted Object Extraction and Region Recomposition: separate object membership from quality level

A user clicks an object in a reference rendered view. Within a small image-space neighborhood, the method finds the nearest positive-depth Gaussian whose projected center meets the distance constraint, lifts the click to its 3D position, and projects that point into other views to prompt SAM. Once SAM produces foreground masks, each Gaussian receives a binary foreground vote according to whether its projected center falls inside each mask. Its foreground score is the mean of these votes over all participating views. Scores strictly above 0.7 indicate foreground, and the remainder is background; the click-search radius is set to 6.

Voting determines Gaussian identity, whereas the quality hierarchy determines the available subset. These decisions can be combined independently. To request foreground level 7 and background level 2, the system takes foreground Gaussians from the level-7 cumulative submodel, background Gaussians from the level-2 cumulative submodel, and merges them for rendering and coding. Background is retained rather than removed, and each new allocation recomposes an existing representation instead of retraining the scene. Object extraction is optional and can be replaced by another 3DGS segmentation method.

This mechanism relies on accurate multi-view segmentation and center projection. It does not guarantee correct boundaries for thin structures or severe occlusions, and the paper does not establish a general system for complex preferences among arbitrarily many objects.

4. Cross-Layer Conditional Coding: use received Gaussians to predict the next layer's attribute distribution

LayeredCGS processes positions, geometry-related attributes, and color-related attributes separately within each layer. Positions are pre-quantized to 16 bits and compressed by RENO-GS, a lightweight occupancy codec based on RENO and accelerated with TorchSparse. A small number of duplicate positions introduced by voxelization are recorded losslessly with DEFLATE. Opacity, rotation, and scale are sensitive to errors, so they are quantized and then losslessly coded. Color follows FCGS's learned mask split between an MLP-based lossy branch and a quantized, losslessly coded branch; the mask is also included in the bitstream. Lossless here refers to the quantized symbols, not exact preservation of the original floating-point scene.

The key cross-layer change is in the attribute entropy model. Already decoded lower-layer Gaussians aggregate latent features with distance-based weights into multiresolution grids: one 3D grid and three orthogonal plane grids, xy, xz, and yz. An undecoded Gaussian queries spatial features at its position. A context analyzer combines these with positional frequency encoding to predict distribution parameters. Spatial context is fused with intra-Gaussian channel context and a hyperprior to form a three-component Gaussian mixture model, supplying means, scales, and mixture weights for entropy coding of geometry and lossless color attributes.

This does not synthesize missing detail at the receiver. Instead, sender and receiver use the same decoded lower-layer Gaussians to predict higher-layer attributes, reducing coding redundancy. Every newly decoded layer provides both additional renderable primitives and context for subsequent layers. That is why truncatable progressive delivery can coexist with cross-layer dependencies.

A Worked Example

Consider kitchen in Fig. 7. Uniform allocation uses foreground level 2 and background level 2, producing a 2.35 MB file. Region-adaptive allocation selects foreground level 4 and background level 1, producing 2.30 MB. The object of interest retains more enhancement layers while the background falls back to the base level; this is not simply a higher global quality setting applied to the same stream.

ROI PSNR improves from 19.65 to 21.81 dB, but full-image PSNR falls from 17.39 to 16.39 dB. Object scalability enables precisely this controlled trade-off. It should not be presented as cost-free enhancement of the whole scene at the same size. This ROI example also differs from the global progressive-decoding example in Fig. 6, so their PSNR values must not be mixed.

Loss & Training

LayeredCGS is trained on approximately 7,000 real scenes from DL3DV-10K, with pretrained Gaussians generated from 960P images and SfM data. Its attribute entropy model minimizes geometry-plus-color attribute bits divided by the number of Gaussians times the attribute dimension, which is 56 by default. Because quantization steps are fixed, this objective excludes a distortion term. A random layer is sampled at each iteration, and all layers share network parameters. This is a separate training process from RecastGS's large-to-small scene distillation.

RENO-GS is trained separately using cross-entropy for occupancy prediction; other relevant modules reuse pretrained FCGS weights. The paper uses Adam for 50 epochs, with a learning rate decaying linearly from 0.0002. The 3D grid resolutions are 60, 80, and 100; the 2D grid resolutions are 300, 400, and 500. Experiments run on an RTX 4090.

Key Experimental Results

Main Results

Evaluation covers Mip-NeRF 360, Tanks & Temples, and Deep Blending, reporting PSNR, SSIM, LPIPS, and BD-BR. BD-BR measures the average rate difference between rate-distortion curves over a shared quality interval; negative values indicate savings. Compression uses file size as the rate axis, whereas representation compaction uses Gaussian count, so these measures are not interchangeable. The abstract summarizes approximately 35% BD-Rate improvement over FCGS. The feed-forward comparison uses the same vanilla 3DGS models, disables distillation and object extraction, and retains layer search.

The following table uses explicit ROI operating points from Fig. 7. Left and right values indicate uniform and region-adaptive allocation, respectively; quality levels are foreground / background.

Scene	Quality Levels	Size MB	Full PSNR dB	ROI PSNR dB	ROI Gain dB
bicycle	3 / 3 to 7 / 2	6.55 to 6.03	19.61 to 18.63	20.69 to 21.82	+1.13
garden	3 / 3 to 6 / 2	7.06 to 6.30	16.85 to 15.28	25.70 to 26.82	+1.12
kitchen	2 / 2 to 4 / 1	2.35 to 2.30	17.39 to 16.39	19.65 to 21.81	+2.16

These points have comparable, not identical, sizes; adaptive allocation is slightly smaller in each case. The final column subtracts the two ROI PSNR values in the source figure. ROI-PSNR BD-BR values are -28.57%, -51.96%, and -33.75%, respectively, while BD-BR based on full-image PSNR worsens by 4.59%, 12.05%, and 7.06%.

Ablation Study

These results come from Fig. 8 and its accompanying discussion on Mip-NeRF 360. Additional reductions retain the paper's wording; percentages with different reference configurations must not be added directly.

Module or Configuration	Reference	Reported BD-BR Improvement	Interpretation
Overcomplete search, no finetuning	GoDe without finetuning	6.06% reduction	Layer selection improves submodels
Small-to-large progressive distillation	Finetuned GoDe baseline	21.22% reduction	Distillation already helps
Large-to-small progressive distillation	Finetuned GoDe baseline	31.27% reduction	The adopted ordering is better
LayeredCGS without cross-layer context or RENO-GS	FCGS	30.07% reduction	Organized layers already aid coding
Add cross-layer context	The preceding layered coding configuration	Additional 8.06% reduction	Exploits cross-layer correlation
Add RENO-GS	Further addition of the geometry codec	Additional 1.25% reduction	Also reports about 1.7x decoding speedup

Key Findings

The kitchen example in Fig. 6 shows an actual preview-to-refinement trajectory. The first row gives base-layer costs; later rows give incremental layer costs. Quality always describes the cumulative model, so incremental sizes should not be mistaken for cumulative file sizes.

Decoded Layers	Current Layer Size MB	Current Layer Decoding s	Cumulative PSNR dB	Cumulative SSIM	Cumulative LPIPS
1	2.534	0.58	27.27	0.8683	0.2180
1-2	+0.610	+0.36	28.19	0.8869	0.1901
1-3	+0.867	+0.45	29.17	0.9017	0.1668
1-4	+1.210	+0.76	29.99	0.9127	0.1484

Layering improves low-capacity submodels and supplies structure for probabilistic coding, but those benefits should be assessed separately. The RecastGS ablation discussion reports approximately 17.4 seconds for layer search and 15 minutes for progressive distillation on average. The method section separately gives 8.3 seconds of additional search overhead relative to GoDe; these are different timing scopes. The 0.58-second figure above is base-layer decoding for this example, not total preparation and transmission latency.

Highlights & Insights

• Object masks and quality levels are orthogonal. An object can receive different cumulative layers without relearning its identity. This composition is better suited to interactive changes than training a separate compression model for every ROI.
• An importance hierarchy can also be an entropy-coding context. Lower layers are useful both for preview and for predicting higher-layer attributes. Representation organization and delivery order jointly affect compression efficiency.
• Search finely, execute coarsely. Overcomplete layering separates the granularity of importance decisions from the final number of operating levels. This is transferable to hierarchical pruning where deployment should not require many more quality settings.

Limitations & Future Work

• Author-stated scope. ROI experiments mainly allocate quality between foreground and background. Complex multi-object preferences, thin structures, and severe occlusions remain open directions, as do anchor-based and pixel-aligned 3DGS formats.
• Per-scene preparation costs matter. The strongest settings can include roughly 15 minutes of progressive distillation and can also replace the reconstruction backbone with GaussianSpa. They do not share the budget of a setting without those optimizations. Deployment evaluation should separate search, distillation, segmentation, encoding, transmission, and decoding time.
• ROI metrics depend on SAM masks. Evaluation regions are generated by SAM; the main text does not establish that they match manually annotated object boundaries. Errors from multi-view center voting near boundaries or occlusions deserve separate measurement. This is an observation about the evaluation design, not an author-reported result.
• Progressive delivery is not arbitrary random access. Bitstream segments have lower-layer context dependencies. The main text does not fully measure the system cost of frequent ROI changes or missing base layers. Explicit dependency management and multi-object bandwidth scheduling are useful next steps.

Related Work & Insights

• GoDe: Supplies gradient-importance layering and cumulative quality levels. This paper adds overcomplete search, large-to-small teacher distillation, and independent object masks rather than inventing the Gaussian hierarchy from scratch.
• FCGS: Supplies feed-forward attribute coding, color routing, channel context, and hyperpriors. LayeredCGS's key differences are ordered cross-layer spatial context and RENO-GS position coding; not all codec components originate in this paper.
• SAM and SAGD: Prompted extraction uses existing 2D segmentation and 3D Gaussian object-extraction ideas. The contribution centers on combining segmentation with quality levels for ROI compression, not introducing a general-purpose segmentation model.
• HAC / ContextGS: These methods use anchor-based representations, unlike the vanilla 3DGS format handled here. Cross-method rate-distortion comparisons measure system performance, not the replacement of one coding module under identical representation and optimization budgets.

Rating

These are the note author's subjective ratings, each out of 5.

• Novelty: 4/5. Object-level recomposition and layered coding work together meaningfully, although several foundations come from GoDe, FCGS, and existing segmentation methods.
• Experimental Thoroughness: 4/5. Three standard datasets, ROI trade-offs, module ablations, and progressive latency are covered; complex multi-object scenes and end-to-end streaming tests remain limited.
• Writing Quality: 4/5. The two-stage architecture and figures are clear, but BD-BR summaries, timing scopes, and segment dependencies require careful distinction.
• Value: 4/5. Useful for progressive viewing and selective transmission of existing 3DGS assets, with practical benefits depending on preparation cost and ROI stability.