AdaptiveSplat: Texture Aware Controllable 3D Gaussian Allocation for Feed-Forward Reconstruction

Conference: ECCV2026
Paper: Official page · PDF
Project: AdaptiveSplat
Area: 3D Vision
Keywords: Gaussian pruning, feed-forward reconstruction, wavelet texture energy, controllable budget, novel view synthesis

TL;DR

AdaptiveSplat uses local texture complexity to decide where to prune 3D Gaussians, then predicts updated attributes through a mask-conditioned head, enabling budget-controlled reconstruction without test-time scene optimization and reaching 20.740 dB PSNR on RE10K after pruning 80% of the primitives.

Background & Motivation

Pixel-aligned feed-forward reconstruction maps each input pixel to a Gaussian, avoiding per-scene fitting but also turning image sampling density into representation density. A plain wall receives roughly the same allocation density as richly textured fabric, although their representational needs differ substantially. Adding input views further enlarges the representation without ensuring that every additional primitive contributes new information.

Simply deleting low-opacity or low-contribution Gaussians does not address the whole problem. Their scales, opacities, and appearance were predicted for a dense collection; after pruning, the survivors may not cover the newly empty regions, producing black patches. Per-scene fine-tuning can partially repair the damage, but sacrifices the latency advantage of feed-forward reconstruction. If an entire local region has disappeared, neighboring Gaussians may expand across inappropriate boundaries and introduce blobby artifacts. Allocation and post-pruning adaptation therefore need to be considered together.

AdaptiveSplat draws on the observation that high-frequency image content is often sparse and localized. Texture energy estimates where detail is needed, while clustering by 3D position and color organizes decisions around local surface regions. Core idea: keep a small set of representatives in low-texture regions, prioritize detailed regions, and feed the pruning mask into the Gaussian head so that surviving primitives learn to cover for their removed neighbors in a single forward pass.

Method

Overall Architecture

The inputs are multi-view RGB images and a user-selected pruning ratio. A pretrained multi-view backbone produces pointmaps and visual features: the sparse-view implementation uses MASt3R, while the dense-view implementation uses VGGT, neither requiring input camera poses. The paper also implements its approach on the pose-conditioned MVSplat backbone. The output is a compact 3D Gaussian representation suitable for novel view rendering, not a repaired 2D image.

Three connected designs drive the pipeline. Wavelet Texture Energy measures local detail in the input images. SuperCluster Budget Pruning aggregates this signal into local 3D regions and generates per-view binary masks. The Mask-Conditioned Gaussian Head combines those masks with backbone features to predict updated attributes for retained primitives. Pointmaps supply positions, masks determine retained indices, and the Gaussian head adapts the remaining representation to its sparse configuration.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multi-view images"] --> Texture["Wavelet Texture Energy"]
    Input --> Backbone["Multi-view backbone<br/>Pointmaps and visual features"]
    Texture --> Pruning["SuperCluster Budget Pruning"]
    Backbone -->|Pointmaps| Pruning
    Budget["User pruning ratio"] --> Pruning
    Pruning -->|Binary masks and retained indices| Head["Mask-Conditioned Gaussian Head"]
    Backbone -->|Visual features| Head
    Head --> Output["Compact Gaussians<br/>Novel view rendering"]

Key Designs

1. Wavelet Texture Energy: turn local detail requirements into a computable signal

A single-level 2D discrete wavelet transform (DWT) decomposes each image into horizontal, vertical, and diagonal high-frequency detail components. Summing their absolute coefficients yields the local texture energy. For input view \(k\), the definition is:

\[ E^{(k)}_{i,j}=\sum_{d\in\{h,v,\mathrm{diag}\}}\left|W^{(k)}_d(i,j)\right|. \]

This energy measures image-frequency responses; it is neither physical energy nor a direct estimate of rendering error. Absolute values prevent cancellation between positive and negative responses, while multiple orientations capture different edges and fine textures. A plain wall generally receives a lower score than patterned wallpaper or fabric, providing an ordering signal for where fewer Gaussians may suffice.

The signal can be computed from input images without first rendering many candidate views to estimate individual Gaussian importance, which makes it compatible with a feed-forward pipeline. However, high image frequency is not the same as geometric complexity: intricate patterns on flat surfaces and specular highlights can both produce strong responses. Subsequent clustering adds spatial structure, but does not fully disentangle texture from geometry.

2. SuperCluster Budget Pruning: retain local representatives instead of emptying entire regions

Pointmaps from all views are aggregated, with every point carrying its 3D coordinates and RGB color. K-means operates in the six-dimensional \((x,y,z,r,g,b)\) space to create \(K\) disjoint SuperClusters. Spatially close points with similar colors tend to be grouped, turning independent pixel-level decisions into allocation over local surface regions. Each cluster is scored by averaging the texture energies of the input pixels associated with its points. The default is \(K=300\).

The user controls the fraction removed, \(\beta\), rather than the fraction retained. In the paper's pixel-aligned formulation, \(m\) views of resolution \(H\times W\) produce \(N=mHW\) candidate primitives, and the target retained count is:

\[ B=\left\lfloor(1-\beta)N\right\rfloor. \]

Thus, \(\beta=0.8\) means keeping approximately 20%, not 80%. According to the operational description in Figure 4, clusters are considered for pruning in increasing order of average texture energy. Within selected low-energy clusters, the highest-energy 10% of primitives are retained, using the default local retention ratio \(\gamma=0.1\); high-energy clusters are retained in full. That 10% rule is local, not a requirement to keep only 10% of the entire scene.

Retained indices are mapped back to the input views to produce pixel-aligned binary masks: one denotes retention and zero denotes removal. These masks determine the output selection and condition the next module. Preserving representatives within a region gives subsequent scale adaptation suitable centers from which to recover coverage, instead of forcing Gaussians on neighboring surfaces to expand into a completely empty region.

A reproducibility caveat matters here. Section 3.2's formulas for the largest selected index, accumulated retained count, and final-cluster remainder do not clearly incorporate Figure 4's statement that high-energy clusters remain intact. They therefore do not provide an unambiguous, complete algorithm for enforcing the budget. This note separates the clear budget objective and illustrated selection principle from unresolved boundary handling; it does not invent rounding, truncation, or final-cluster allocation rules. The normalization of coordinates relative to RGB values in six-dimensional clustering is also not specified in the available main text.

3. Mask-Conditioned Gaussian Head: tell surviving primitives which neighbors have disappeared

Predicting dense attributes with a fixed head and then deleting masked entries leaves survivors with scales and opacities suited to the original dense representation. AdaptiveSplat instead uses a DPT-based Gaussian head with additional convolutional layers that fuse multi-view ViT features and binary masks. For retained indices, it predicts anisotropic scales, rotation quaternions, opacity, and spherical harmonics appearance coefficients. The head's stated output does not include center positions, which come from the pointmaps; it should not be described as reoptimizing all geometric coordinates.

The mask is more than a final zero-or-one multiplier. It tells the predictor which primitives remain in the local neighborhood, allowing identical visual features to produce different attributes under different pruning layouts. A Gaussian that originally covered a small area can expand when nearby representatives disappear. Detailed regions retain more centers and need not rely on a few large primitives to cover fine texture. Figure 9's scale-distribution analysis supports larger Gaussians in low-texture regions, but the method does not prescribe a single explicit expansion factor.

This distinction also clarifies the ablations. A fixed Gaussian head tests whether attributes must be learned for the pruned representation. Removing the mask tests whether a trained head still needs explicit knowledge of the pruning layout. These are different questions. The conditional head requires offline training, but does not require iterative fitting for each new scene at inference.

A Worked Example

Consider two indoor images containing a plain wall and a textured sofa. This is an illustrative input, not an additional experiment. Setting \(\beta=0.8\) requests a total of \(\lfloor0.2N\rfloor\) retained primitives. The wavelet transform assigns relatively low energy to the wall and higher energy to the fabric. Six-dimensional clustering then groups nearby, similarly colored points, rather than merging unrelated surfaces solely because of their locations in a 2D image.

Selected low-energy wall clusters keep a small set of representatives, while detail-rich clusters receive priority. If a cluster undergoing the full local 10% rule contains exactly 100 candidates, it retains the 10 with the highest texture energies. This arithmetic illustrates the local rule only, not the unresolved global budget boundary treatment. Masks and visual features then enter the Gaussian head together: wall representatives can broaden their coverage, while denser sofa Gaussians preserve texture, after which the representation directly renders the target view.

Loss & Training

Training samples pruning ratios from \(\beta\sim U[0,1)\) so that one head encounters different sparsity levels instead of overfitting a single budget. Supervision comes from target camera views: the retained Gaussians render an image that is compared with the ground-truth target using a pixel reconstruction term and LPIPS perceptual regularization. The text gives a perceptual weight of \(\lambda=0.001\). Operators and formatting in cached Equation (1) are corrupted, so this note reports the verifiable loss components without reconstructing an uncertain full equation.

The paper reports one NVIDIA A100 40GB GPU and 48 GPU hours of training per dataset. RE10K, ACID, and DL3DV follow the stated train-test protocols. Sparse-view evaluation uses two input images, while dense-view evaluation uses 6, 9, 16, and 32 images. The inference section explicitly gives \(\beta\in[0,0.8]\); separate analysis mentions 0.9, but that does not establish it as a systematically validated default range across every dataset.

Key Experimental Results

Main Results

The following selections come from Tables 1 and 2. PSNR is measured in dB and higher is better; lower LPIPS and higher SSIM are better. Section 5.1 states that the external pruning baselines shown here allow per-scene fine-tuning, whereas AdaptiveSplat itself retains feed-forward inference. These are practical comparisons under pruning constraints, not controlled ablations that change only a pruning module while holding training conditions identical.

Dataset / input	Method	Pruning ratio	PSNR	LPIPS	SSIM
ACID / 2 views	HiSplat + LightGaussian	0.4	19.785	0.604	0.566
ACID / 2 views	AdaptiveSplat	0.4	22.549	0.299	0.640
ACID / 2 views	AdaptiveSplat	0.8	21.055	0.342	0.593
RE10K / 2 views	HiSplat + LightGaussian	0.4	16.598	0.596	0.558
RE10K / 2 views	AdaptiveSplat	0.4	22.294	0.235	0.735
RE10K / 2 views	AdaptiveSplat	0.8	20.740	0.272	0.692
DL3DV / 6 views	AnySplat + LightGaussian	0.4	11.361	0.671	0.262
DL3DV / 6 views	AdaptiveSplat	0.4	20.448	0.334	0.601
DL3DV / 6 views	AnySplat + LightGaussian	0.8	9.169	0.697	0.177
DL3DV / 6 views	AdaptiveSplat	0.8	20.049	0.408	0.556

Increasing pruning from 40% to 80% lowers RE10K PSNR from 22.294 to 20.740 and six-view DL3DV PSNR from 20.448 to 20.049. Sparsity is not free, but degradation is relatively gradual for this method. DL3DV LPIPS nevertheless rises from 0.334 to 0.408, so a modest PSNR decline must not be interpreted as no loss of perceptual detail.

Ablation Study

Table 3 fixes \(\beta=0.4\). Each result cell lists PSNR / LPIPS / SSIM in that order. Precision follows the ablation table rather than mixing its rounded values with the main comparison.

Configuration	ACID	RE10K	What it tests
Without texture ranking	21.31 / 0.326 / 0.623	20.73 / 0.260 / 0.713	Random cluster selection for pruning, with model training
Without adaptive GS head	17.44 / 0.445 / 0.490	19.81 / 0.399 / 0.617	Direct selection from fixed pretrained head predictions
Without binary mask input	22.52 / 0.318 / 0.642	20.06 / 0.339 / 0.656	Trained head without explicit pruning layout
Full model	22.55 / 0.299 / 0.639	22.29 / 0.235 / 0.735	Texture ranking, attribute adaptation, and masks

The full model's ACID SSIM is 0.639 in Table 3 but 0.640 in Table 1. Each is preserved as reported rather than silently reconciled. Removing the mask actually raises ACID SSIM to 0.642, so the evidence does not support claiming that every component improves every metric.

Key Findings

• Attribute adaptation has the strongest individual evidence: removing the adaptive head reduces PSNR by 5.11 dB on ACID and 2.48 dB on RE10K. Better selection alone is insufficient.
• Mask benefits depend on the dataset: removing it costs 2.23 dB on RE10K but only 0.03 dB on ACID, although ACID LPIPS worsens from 0.299 to 0.318. The benefit is not uniform.
• Table 4 reports PSNR of 22.549 at \(K=300\) and 22.559 at \(K=400\), indicating diminishing returns. Retention ratios \(\gamma=0.05/0.10/0.15/0.20\) yield 21.796/22.549/21.587/22.262, respectively; 0.10 is best among these settings, but the trend is not monotonic.
• Rendering throughput is measured by synthesizing 1000 novel views at each pruning strength. The figure reports increasing FPS with stronger pruning, but the cache supplies no reliably readable FPS or latency values, so no numerical speedup is asserted here.

Highlights & Insights

• Pruning and attribute adaptation belong together. Surviving Gaussians with fixed centers can adapt scales, opacity, and appearance to a new layout, making compression possible without a separate per-scene repair phase.
• Explicit binary masks connect budget control to visual prediction. They describe where coverage must change more precisely than a single global sparsity value, with the RE10K ablation supporting this role.
• Region-level allocation adds structure beyond a global ranking of independent primitives. Preserving local representatives before adapting their attributes is a promising idea for other sparse representations, but such transfer is not established by this paper.

Limitations & Future Work

• The authors acknowledge failures on smooth surfaces with high-frequency patterns and on specular highlights. Texture energy alone cannot decide whether complex appearance requires more geometry or a stronger appearance model; joint texture-and-geometry reasoning remains necessary.
• Reproducibility details remain incomplete. The budget-boundary formulas do not fully align with Figure 4's retention description, and the available main text does not specify coordinate/color normalization. Both can affect clustering and the actual retained count.
• Broad evaluation claims should not be confused with fully verified results in this note. DTU zero-shot and Tanks and Temples results are deferred to supplementary material, while the available cache contains the main paper and references only; no scores for them are reported here.
• Fewer primitives do not imply a proportional reduction in end-to-end GPU memory. The authors report the same GPU memory load as the base model, and dense feature extraction, clustering, and mask construction remain. One-time reconstruction cost, peak memory, and repeated rendering cost should be measured separately.
• Useful follow-up evaluations would add geometric confidence and cross-view consistency signals, explicit budget-enforcement pseudocode, end-to-end latency on matched hardware, and unpruned baseline curves. These are suggestions from this note, not completed experiments claimed by the authors.

Related Work & Insights

• Relative to NoPoSplat, HiSplat, and AnySplat: these methods supply feed-forward Gaussian representations from sparse or dense views. AdaptiveSplat mainly addresses explicit representation-density control rather than introducing a new foundational multi-view geometry backbone.
• Relative to EAGLES, LightGaussian, and PUP-3DGS: these compression methods select primitives through contribution, importance, or uncertainty, whereas AdaptiveSplat combines texture-organized regional allocation with a trained adaptive head. The reported comparisons concern their transfer into the paper's feed-forward setting, not their overall strength in their original per-scene reconstruction regimes.
• Connection to WaveNeRF: both exploit wavelets to describe frequency information, but AdaptiveSplat uses this information to control Gaussian budgets and pruning layouts. Its key contribution is combining frequency cues with conditional attribute prediction, not merely adding DWT.

Rating

• Novelty: 4/5. Texture, local representatives, and mask-conditioned prediction form a coherent approach to controllable feed-forward sparsification rather than a new backbone.
• Experimental Thoroughness: 4/5. The main paper spans datasets, budgets, and component ablations, but supplementary results are unavailable in this cache and end-to-end cost evidence remains incomplete.
• Writing Quality: 3/5. Motivation and visual explanations are clear, but the budget rule is ambiguous and individual main-table and ablation values differ slightly.
• Value: 4/5. Compact representations without test-time optimization are useful for repeated rendering; deployment benefits still require accounting for backbone and clustering costs.