Neural Harmonic Textures for High-Quality Primitive Based Neural Reconstruction¶

Conference: ECCV 2026
Paper: ECCV page
Code: https://research.nvidia.com/labs/sil/projects/neural-harmonic-textures/
Area: 3D Vision
Keywords: 3D Gaussian Splatting, neural reconstruction, harmonic basis, deferred shading, novel view synthesis

TL;DR¶

The paper attaches learnable feature vectors to the virtual tetrahedron bounding each primitive (Gaussian or triangle), barycentrically interpolates them at ray–primitive intersections, applies sin/cos periodic activations, and then alpha-composites along the ray — turning the blend itself into a harmonic decomposition — so that a single image-space pass of a shallow MLP decodes the final color; it surpasses 3DGS, 2DGS, and Spherical Voronoi at 140+ FPS on MipNeRF360, Tanks & Temples, and Deep Blending (28.74 PSNR on MipNeRF360, 25.68 on Tanks & Temples) and plugs into 2DGS and Triangle Splatting unchanged.

Background & Motivation¶

Since the introduction of 3D Gaussian Splatting, Lagrangian primitive-based representations have largely displaced Eulerian continuous neural radiance fields. The shift stems not only from much faster rendering but also from the structural advantages of explicit representations: primitives adapt to scene detail, scale more gracefully, and readily support motion, deformation, and editing, while aligning well with feed-forward reconstruction pipelines and point-map representations. Yet the limited expressive power of individual primitives remains a fundamental bottleneck. In 3DGS, geometry and appearance are tightly coupled inside each primitive, so representing high-frequency spatial detail requires increasing the number of primitives, which directly inflates memory and slows rendering. Directional appearance is constrained in the same way: view-dependent effects are typically modeled with low-order spherical harmonics, which are spectrally band-limited and poorly suited to sharp specular highlights. Alternative angular parameterizations such as spherical Gaussians, spherical Beta kernels, and Spherical Voronoi raise the directional bandwidth, but they leave the fundamental coupling between spatial support and appearance modeling intact.

The other route is the neural field, which combines positional encodings (Fourier features, multi-resolution hash grids) with an MLP decoder to achieve very high local representational capacity. Recent work has tried to bridge the two paradigms by treating primitives as acceleration structures that query a global neural field (e.g., Radiance Meshes, NeST). Such hybrids, however, inherit the key limitations of global neural representations: features stay globally anchored (e.g., on a hash grid), do not follow the primitives under motion or deformation, and scale poorly to large scenes and high-frequency detail, which also erodes their editability.

The angle taken here is that primitives themselves can serve as positional encodings. The authors anchor learnable features on a virtual scaffold enclosing each primitive and interpolate them locally at ray–primitive intersections, which makes the features Lagrangian: they move, deform, and disappear with the primitives, without any global spatial structure. That alone is not enough, though. Prior work with interpolated features either decodes appearance per primitive before volume rendering, requiring dozens of MLP evaluations per ray, or — as in 3DGS — approximates view-dependent color once per primitive, which assumes the signal does not vary spatially within a primitive and therefore contradicts the very intra-primitive texture this paper targets. Instead, the authors borrow from Fourier analysis, where a complex signal is expressed as a sum of harmonic components (periodic functions of different amplitudes), and apply periodic activations (sine and cosine) to the interpolated features before compositing them along the ray, then accumulate them weighted by opacity. Core idea: treat the interpolated features on a primitive's virtual scaffold as a local positional encoding, turn them into superposable harmonic components via sin/cos activations, and let alpha compositing itself perform the signal synthesis, which reduces neural decoding to a single evaluation per pixel.

Method¶

Overall Architecture¶

The method is presented on 3D Gaussian primitives following the 3DGUT formulation, and it targets a single problem: how to raise the expressivity of an individual primitive — especially spatial variation and high-frequency appearance inside the primitive — without increasing primitive counts and without any global feature grid, while keeping rendering real-time. The pipeline is: for each Gaussian, define a virtual tetrahedron in whitened canonical space and assign one learnable feature vector to each of its four vertices; when a ray hits a primitive, take the point of maximum projected-Gaussian response along the ray and barycentrically interpolate the vertex features to obtain the local feature \(f_i\); apply sin/cos periodic activations to the interpolated feature so it becomes a set of harmonic components, and accumulate them along the ray weighted by opacity and transmittance; concatenate the accumulated harmonics with a ray-direction encoding and decode the pixel color with one shallow MLP in a single image-space pass. The neural network is touched only in the last step; rasterization, interpolation, and blending all happen inside the existing primitive rendering pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["ray + primitive set"] --> B["Primitive-bound feature embedding<br/>tetrahedron vertex features<br/>barycentric interpolation at hit"]
    B --> C["Harmonic texturing<br/>sin/cos periodic activation<br/>opacity-weighted accumulation"]
    C --> D["Neural deferred shading<br/>single image-space MLP pass"]
    D --> E["RGB pixel"]

Key Designs¶

1. Primitive-bound feature embedding: making the primitive itself a local positional encoding

Spatial structures such as triplanes, voxel grids, and multi-resolution hash grids are effective but globally defined regular grids: they scale poorly to large scenes and high-frequency detail, and because they are fixed in space they fail to follow the geometry under motion, deformation, or editing. The paper instead exploits the Lagrangian nature and adaptivity of primitive-based representations. The bounded-support isosurface of an anisotropic 3D Gaussian becomes a sphere in whitened canonical space, so the authors define the virtual tetrahedron bounding that sphere and assign an \(N_f\)-dimensional learnable feature vector \(f^j\) to each of its four vertices. For every ray intersecting the primitive, they take the point \(p^*\) of maximum projected-Gaussian response along the ray — as in 3DGUT — and barycentrically interpolate the four vertex features into the local feature \(f\) at that point (a detailed derivation is in the supplementary material). The result behaves like a continuous, spatially varying texture inside the primitive: the larger the differences among vertex features, the more rapidly the texture varies in space. Unlike global feature grids, this needs no auxiliary spatial data structure, and the features rotate, scale, translate, and vanish together with the primitive, so motion, deformation, and scene editing are supported natively — exactly the capability that hybrid "primitives plus global neural field" pipelines give up.

2. Harmonic texturing: turning alpha compositing into a weighted sum of harmonic components

Decoding features into appearance per primitive before volume rendering costs dozens of MLP evaluations per ray; 3DGS reduces color-decoding overhead by evaluating view-dependent color once per primitive, cutting complexity from the number of ray–primitive intersections to the number of primitives, but that approximation assumes the signal does not vary spatially within a primitive and is thus unsuitable here. The authors defer decoding and instead change the nature of what is blended: before compositing along the ray, they apply periodic functions (sine and cosine) to the interpolated features, so the alpha accumulation becomes a weighted sum over components at different frequencies — what they call Harmonic Textures. Two roles are worth naming. The interpolation function acts as a frequency modulator: the larger the differences between vertex features within a primitive, the faster the activated components oscillate and the higher the spatial frequency that can be represented. The kernel-weighted opacity of each primitive acts as the harmonic amplitude: the more opaque the primitive, the more that frequency contributes to the final pixel (Fig. 4 visualizes the harmonics without amplitude weighting for clarity). This reformulation matters because it moves representational burden out of the network and into the features and the blending operator: after compositing, the signal in image space is already a high-dimensional, frequency-rich implicit representation, and all that remains is to decode it once.

3. Neural deferred shading: cutting neural inference from dozens of evaluations per ray to one per pixel

Since the harmonics accumulated along the ray are already rich, decoding can be extremely cheap: the accumulated harmonics are concatenated with the ray direction encoded with second-degree spherical harmonics \(\mathrm{SH}_2(\mathbf{d})\in\mathbb{R}^9\), following Instant-NGP, and a shallow MLP outputs the pixel color \(c\) directly with no further positional encoding. The rendering equation is

\[c = \mathrm{MLP}_{\theta}\Big(\sum_{i\in\mathcal{G}} T_i\,\alpha_i\big[\sin(\mathbf{f}_i);\ \cos(\mathbf{f}_i)\big],\ k\cdot \mathrm{SH}_2(\mathbf{d})\Big)\]

where \(\mathbf{f}_i=\mathrm{interpolate}(f_i^0,f_i^1,f_i^2,f_i^3;\,p_i^*)\) is the feature interpolated at the maximum-response point of the \(i\)-th primitive, \(\mathcal{G}\) is the set of primitives intersected by the ray, and \(\alpha_i\) and \(T_i\) are the opacity and accumulated transmittance defined as in Eq. (2). ⚠️ Eq. (3) of the source is garbled in the cached text; its structure was reconstructed from context here, so the exact symbols and concatenation order should be checked against the original paper.

Because the MLP is evaluated once per pixel and can be kept tiny (the main experiments use 128 wide × 3 hidden layers), the method preserves the efficiency of deferred shading without shifting the burden onto the network the way global feature grids do. In practice the authors implement custom CUDA kernels for the forward and backward logic on top of gsplat with the 3DGUT formulation, use half-precision memory fetches for feature vectors during rasterization to reduce register pressure, and use tiny-cuda-nn's JIT-compiled cooperative vector MLPs trained and evaluated in FP16, with automatic loss scaling (standard mixed-precision practice) for half-precision stability.

A Worked Example¶

Take the headline configuration: 64 features per primitive (16 per tetrahedron vertex). A ray traverses the scene and hits a sequence of Gaussian primitives. For each hit, the rasterizer finds the maximum-response point \(p_i^*\) of the projected Gaussian, barycentrically interpolates the 16-dimensional feature \(\mathbf{f}_i\), and activates it with sine and cosine to form a 32-dimensional harmonic vector. That vector is weighted by \(T_i\alpha_i\) and accumulated along the ray together with the components of the other primitives, yielding a 32-dimensional harmonic sum. The ray direction is separately encoded into 9 second-degree spherical harmonics; concatenated with the harmonic sum, it feeds a 128-wide, 3-hidden-layer MLP that produces this pixel's RGB in one forward pass. However many primitives the ray hits, the neural network runs exactly once. ⚠️ The dimension bookkeeping (32 + 9) is derived here from Eq. (3) and the reported configuration; the source equation is garbled in the cached text, so refer to the original paper.

Loss & Training¶

Optimization follows 3DGS-MCMC entirely: the densification strategy, loss, and regularization terms are unchanged. The regularizer constrains both primitive opacity and scale,

\[\mathcal{L}_{reg} = \frac{1}{P}\sum_{i=1}^{P}\sigma_i + \frac{1}{P}\sum_{i=1}^{P}\lVert \mathbf{s}_i \rVert\]

and the total loss is \(\mathcal{L} = (1-\lambda)\mathcal{L}_1 + \lambda\,\mathcal{L}_{\text{D-SSIM}}\), where \(P\) is the number of primitives and \(\mathbf{s}_i\) the scale vector of primitive \(i\). Three adjustments are made for the newly introduced parameters: a cosine annealing scheduler for the feature and MLP learning rates (mirroring the exponential schedule applied to primitive positions in 3DGS); an EMA filter on the MLP weights \(\bar{\theta}_t \leftarrow \gamma\,\bar{\theta}_{t-1} + (1-\gamma)\,\theta_t\) for robustness to noise and to avoid overfitting individual frames; and a final 3000 iterations that optimize only the features and MLP weights with all regularization disabled and all other parameters frozen — a "color refinement" step that slightly improves color fidelity, particularly in large-scale scenes. Other hyper-parameters are in the supplementary material.

Key Experimental Results¶

Main Results¶

Results are reported on MipNeRF360, Tanks & Temples, and Deep Blending. Two evaluation-protocol details matter: on MipNeRF360 all methods train and evaluate directly on the officially provided JPEG reference images (disabling gsplat's default downscaling) to compare properly with prior work; and the method uses 2M primitives indoors and 5M outdoors, training indoor scenes for 45k iterations and outdoor for 25k. The table below excerpts representative rows.

Method (category)	MipNeRF360 PSNR↑ / SSIM↑ / LPIPS↓	Tanks & Temples	Deep Blending
Mip-NeRF 360 (pure neural field)	27.60 / 0.788 / 0.275	22.22 / 0.754 / 0.290	29.40 / 0.899 / 0.306
ZipNeRF (hash-encoded neural field)	28.55 / 0.829 / 0.218	23.64 / 0.836 / 0.179	–
3DGS-MCMC (primitive + SH)	27.99 / 0.830 / 0.229	24.46 / 0.866 / 0.174	29.49 / 0.912 / 0.306
2DGS (primitive + SH)	27.22 / 0.804 / 0.275	22.85 / 0.827 / 0.244	29.56 / 0.904 / 0.325
Spherical Voronoi (primitive + strong angular basis)	28.56 / 0.835 / 0.228	24.80 / 0.871 / 0.172	30.34 / 0.914 / 0.299
Neural Shell Texture Splatting (primitive + neural field)	26.54 / 0.776 / 0.260	–	–
NHT (Ours)	28.74 / 0.834 / 0.216	25.68 / 0.882 / 0.141	30.94 / 0.919 / 0.302

The method beats prior real-time and offline work on all three datasets: on MipNeRF360 it is 0.18 dB above Spherical Voronoi and 0.19 dB above ZipNeRF while rendering in real time, and it leads on LPIPS on Tanks & Temples and Deep Blending as well (Deep Blending's 0.302 LPIPS is marginally above SV's 0.299, a rare exception).

Because Table 1 mixes different training schedules, primitive counts, and parameter budgets, the authors run a controlled experiment in which all methods live in the same framework (gsplat) with a 1M primitive cap, 30k iterations, the same appearance parameter budget per primitive (48), and identical per-scene hyper-parameters — the only variable is the appearance model. NHT still beats both regular SH and the current state-of-the-art angular basis Spherical Voronoi while using less storage; the only cost is frame rate dropping from 200+ to 140, which remains real time.

Appearance model (all with MCMC, 1M primitives / 30k iterations)	MipNeRF360 PSNR↑ / LPIPS↓ / FPS↑	Tanks & Temples	Deep Blending
3DGS + SH	27.94 / 0.246 / 251	24.25 / 0.188 / 294	29.98 / 0.317 / 331
3DGUT + SH	27.93 / 0.247 / 201	23.99 / 0.192 / 245	30.21 / 0.318 / 282
3DGUT + Spherical Voronoi	28.15 / 0.248 / 202	24.18 / 0.187 / 242	30.29 / 0.320 / 267
3DGUT + NHT (Ours)	28.46 / 0.232 / 140	24.79 / 0.169 / 226	30.88 / 0.311 / 240

In this setting NHT trains to 30k iterations in roughly 14.5 minutes on MipNeRF360 on average (1M primitives, RTX 5090).

Two further applications are shown. The first is semantic scene reconstruction: expanding only the MLP decoding head fits signals of arbitrary dimension while the remaining layers and the feature size stay the same, forcing the model to exploit correlations across dimensions. The task is joint reconstruction of RGB radiance and LSEG 512-wide semantic features.

Method	RGB PSNR↑ / SSIM↑ / LPIPS↓	Semantic PSNR↑ / cosine sim.↑	FPS↑
Feature 3DGS (176 features/primitive + 512×128 CNN, 1.5× primitives, ~3× memory)	26.27 / 0.785 / 0.262	43.68 / 0.985	3.02
NHT (Ours, 80 features/primitive + 128×3 MLP)	28.16 / 0.824 / 0.228	46.90 / 0.993	28.11

The second is high-resolution 2D image fitting: a new curated dataset of 45.7 MP, 14-bit HDR RAW images (averaged over 15 images) is fitted and compared against Instant NGP and the HDR-capable JPEG-XL encoder. Adapting to 2D requires two changes — a connected 2D triangle mesh topology that is fully opaque (no alpha blending, no kernel, purely interpolation) and Clough-Tocher cubic interpolation instead of barycentric interpolation so that features stay continuous across triangle edges, removing high-frequency artifacts.

Ratio	Method	PSNRµ↑	PSNRtm↑	SSIMtm↑	LPIPStm↓
10×	JPEG-XL	45.31	44.66	0.994	0.001
10×	Instant NGP	37.63	38.91	0.949	0.061
10×	NHT (Ours)	37.16	39.87	0.963	0.023
100×	JPEG-XL	38.53	35.54	0.971	0.017
100×	Instant NGP	35.17	36.39	0.923	0.078
100×	NHT (Ours)	35.06	36.43	0.927	0.048

Ablation Study¶

The most informative ablation reported in the main text is swapping the appearance model into different primitive types, demonstrating primitive-agnosticism. Adaptation is cheap: for 2D primitives the bounding tetrahedron becomes a virtual triangle, or features are anchored directly to explicit triangle geometry, so each primitive carries three feature vectors instead of four.

Config (MipNeRF360)	PSNR↑	SSIM↑	LPIPS↓	Note
3DGUT-MCMC (SH)	27.93	0.828	0.247	baseline
3DGUT-MCMC + NHT	28.46	0.830	0.232	+0.53 dB
2DGS (SH)	27.48	0.816	0.263	baseline
2DGS + NHT	28.27	0.820	0.238	+0.79 dB
Triangle Splatting	27.00	0.808	0.231	baseline (authors state this is a proof-of-concept, not extensively optimized)
Triangle Splatting + NHT	27.52	0.807	0.227	+0.52 dB

The main text also analyzes primitive count: sweeping from 1K to 4M primitives, the method improves over 3DGS-MCMC and 3DGUT-MCMC across the entire range, with the largest gains in the low-primitive regime (≤100k), where the deltas exceed 2 dB — matching prior methods' 1M-primitive quality with roughly a third of the count, and reaching +3.7 dB PSNR on average over 3DGUT-MCMC on the bonsai scene with only 10K primitives.

⚠️ For finer-grained ablations (individual contributions of the training strategies, the choice of feature encoding function, per-primitive feature dimension, and MLP architecture), the main text reports only the conclusions and places the numbers in the supplementary material (Tabs. S10 / S11 / S13 / S14): harmonic functions are reported to be the best feature encoding, and feature dimension and MLP architecture trade quality against speed continuously. This note cannot report those specific numbers.

Key Findings¶

The gain comes from appearance expressivity, not geometry: in the controlled experiment all methods share the same densification and regularization, leaving the appearance model as the only variable, yet NHT still leads by 0.3–0.6 dB, so the improvement is not an artifact of different training schedules.
The low-primitive regime benefits most: the fewer the primitives, the more intra-primitive spatial variation each must carry, and the more valuable harmonic texturing becomes (deltas above 2 dB at ≤100k primitives), with curves converging as counts grow. This directly answers the motivating pain point that high-frequency detail can only be bought with more primitives — it shifts the whole memory–speed–quality trade-off curve.
The cost is a modest frame-rate drop: from 200+ FPS to 140 FPS in the controlled setting, still real time; the authors list being slightly slower than pure 3DGS as an explicit limitation.
High-dimensional signals extend almost for free: widening only the decoding head fits 512-dimensional semantic features with almost no loss in RGB quality (the authors attribute this to high correlation among feature dimensions or to spare bandwidth in the RGB task), at roughly one third of Feature 3DGS's memory and about 9× its frame rate.
Perceptual quality on HDR image fitting clearly beats Instant NGP: at 100× compression LPIPS drops from 0.078 to 0.048 (a 38% relative improvement), though pixel error (PSNRµ) is slightly below Instant NGP, and both lag JPEG-XL substantially.

Highlights & Insights¶

Reinterpreting "blending" as "signal synthesis": the Fourier-series analogy is not rhetoric — it dictates an implementable operator order (periodic activation first, blending second) and incidentally explains the two roles, frequency modulation (vertex-feature differences drive spatial oscillation) and amplitude (kernel-weighted opacity). This way of remodeling one step of a rendering pipeline through a classical signal-processing decomposition transfers to any representation that combines local features with per-ray accumulation.
Geometry and appearance are decoupled without introducing global structure: primitive density can follow geometric complexity instead of being inflated to reproduce high-frequency appearance, which improves both quality and memory efficiency. Because features move with primitives, the method is more natural than "primitive + hash grid" hybrids for editing, deformation, and 4D scenes.
Deferred shading reduces neural inference to one evaluation per pixel: the accumulated harmonics already provide a rich high-dimensional input, so the MLP needs no positional encoding and needs no perceptual-loss supervision or guiding network to stabilize training (unlike INPC-style methods that require NeRF-style rendering guidance). It is a clean example of representational design saving optimization effort.
Primitive-agnosticism: replacing tetrahedra with triangles and four features with three is enough to attach the mechanism to 2DGS and Triangle Splatting, showing it barely depends on any special property of 3D Gaussians.
Engineering details worth noting: half-precision feature fetches during rasterization to reduce register pressure, cooperative vector MLPs, and automatic loss scaling for FP16 stability are what make "dozens of features per primitive with full quality" run inside a real-time pipeline.

Limitations & Future Work¶

The authors acknowledge that the method focuses on per-primitive expressivity, which has inherent downsides: with very sparse supervision it can overfit individual views and lose novel-view synthesis quality, and the extra neural decoder makes rendering slightly slower than pure 3DGS (measured at 140+ FPS).
My own observations: the main text relegates the finest-grained ablations (feature encoding function, feature dimension, MLP architecture, individual training strategies) to the supplementary material and reports only conclusions, so a reader cannot tell how much the harmonic activation actually buys over alternatives; Table 1 compares across different training schedules and parameter budgets (partially remedied by the controlled Table 2), so the "beats offline methods" claim deserves a protocol caveat.
In the HDR image-fitting setting, NHT does not beat Instant NGP on pixel error, so its perceptual advantage and its metric advantage disagree; the authors explain JPEG-XL's lead by its human-perception inductive biases, which means the verdict on this task depends on which metric is emphasized.
Extensions (some proposed by the authors): extracting automatic level of detail from a finer-grained harmonic decomposition; applying the representation to radiance caching, neural physically based rendering, and geometric reconstruction; and removing kernel functions entirely, since derivatives can still be propagated to explicit geometry without a spatially decaying opacity function — which could yield large performance gains, though it would also require rethinking the current "opacity as harmonic amplitude" interpretation.

vs 3DGS / 2DGS / Triangle Splatting: these use low-order spherical harmonics for view-dependent color, which limits directional bandwidth and couples geometry with appearance; this paper leaves the geometry pipeline untouched and replaces only the appearance model, gaining quality at equal primitive count and leading by a wide margin at low counts. The cost is a few extra feature dimensions per primitive and one shared MLP evaluation per pixel.
vs Spherical Voronoi / spherical Gaussians / Beta Splatting: these raise directional bandwidth through stronger angular parameterizations but keep spatial support and appearance coupled inside the primitive; this paper instead builds a local positional encoding in space with a shared neural decoder, and the two are orthogonal in principle, leaving room for combination.
vs Radiance Meshes / Neural Shell Texture Splatting / INPC: they use primitives as acceleration structures for querying a global neural field (hash grid), so features are globally anchored, do not follow motion or deformation, and inherit the scaling problems of Eulerian encodings; this paper anchors features on the primitives themselves (Lagrangian) and, because harmonic blending is already rich, reduces neural inference to one evaluation per pixel.
vs Textured Gaussians / BBSplat and other per-primitive texture methods: they add highly expressive textures (even spatially varying opacity) to each primitive, but optimization becomes harder and prone to overfitting, with substantial storage cost and slower rendering; this paper replaces "heavy primitives" with "lightweight shared MLP plus local features," placing expressivity in low-dimensional features and the blending operator.
vs Feature 3DGS: it also attaches high-dimensional features to primitives, but relies on large explicit high-dimensional features plus a CNN for upsampling; this paper decodes with one small MLP, using roughly one third of the memory and about 9× the frame rate while improving both semantic and RGB metrics.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Reinterpreting alpha compositing as harmonic synthesis and thereby reducing neural decoding to one evaluation per pixel is a simple but effective change of perspective.
Experimental Thoroughness: ⭐⭐⭐⭐ Three standard datasets plus a controlled comparison, a primitive-count sweep, and two extra tasks, though fine-grained ablations are confined to the supplementary material.
Writing Quality: ⭐⭐⭐⭐ The motivation chain (coupling → limits of global grids → the Fourier analogy) is clear, but formulas are sparse and the presentation of Eq. (3) is not reader-friendly.
Value: ⭐⭐⭐⭐⭐ The method is primitive-agnostic and drop-in, directly pushing the quality–memory trade-off curve of real-time novel view synthesis and informing both future primitive representations and deferred-shading pipelines.