Heat Kernel Textures -- the Geodesic Gaussians That Do Not Splat

Conference: ECCV2026
Authors: Simone Foti, Caner Korkmaz, Stefanos Zafeiriou, Tolga Birdal
Paper: ECCV Paper
Area: 3D Vision
Keywords: heat kernel textures, UV-free representation, anisotropic diffusion, Riemannian optimization, differentiable rendering

TL;DR

HKTex places learnable anisotropic heat kernels directly on triangle-mesh surfaces and combines surface-constrained optimization with differentiable ray tracing to replace UV texture storage, recovering detail with a smaller reported storage footprint but without the fast rasterization advantage of Gaussian Splatting.

Background & Motivation

Conventional texturing unfolds a three-dimensional surface into a two-dimensional atlas and reads colors through per-vertex UV coordinates. Its cost includes not only image pixels but also UV coordinates, duplicated vertices at seams, and empty atlas regions that correspond to no surface. Unwrapping complex surfaces also introduces seams, distortion, and uneven texel resolution, so shrinking the image does not eliminate the overhead of the complete texture representation. Vertex colors remove the atlas but tie appearance detail to geometric sampling density; a geometrically flat region with complex colors can still require many additional vertices.

Neural texture fields continuously predict surface color through a network, avoiding an explicit atlas while incurring per-point evaluation and network-storage costs. 3D Gaussian Splatting suggests another direction: allocate explicit primitives adaptively to content instead of filling a fixed regular grid. However, Euclidean Gaussians do not naturally bend along curved surfaces, and direct updates to their three-dimensional positions can move their centers off the mesh. Surface-alignment regularizers impose external constraints, but the primitives' spatial support and optimization paths are still not intrinsically defined on the surface.

The paper therefore targets albedo texture representation on a given mesh, rather than jointly reconstructing geometry, materials, and illumination. A heat kernel describes how heat from a point source diffuses over a surface: it corresponds to a Gaussian in Euclidean space, while general surface geometry determines its shape on a manifold. If primitive definition, movement, splitting, and querying all respect that surface, UV parameterization is no longer needed to connect geometry and appearance. Core Idea: encode color with anisotropic surface heat kernels, allocate texture freedom to continuous surface positions and adjustable kernel shapes, and evaluate the representation directly at ray intersections.

Method

Overall Architecture

The inputs are a known triangle mesh and either an existing UV albedo texture or multi-view supervision images with camera poses. The output is a collection of colored heat-kernel parameters and a global base color, which the renderer uses to evaluate albedo at arbitrary surface points. Mesh geometry is an input condition in this task, not something these kernels replace or primarily optimize. The pipeline comprises spectral precomputation and alignment, continuous local kernel evaluation, geodesic optimization and density control, and intersection-based texture rendering. During training, the final stage produces errors that update kernel parameters; at inference time, parameters are fixed and only local queries and rendering remain.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Mesh["Known triangle mesh"] --> Basis["Spectral Precomputation<br/>and Alignment"]
    Basis --> Eval["Continuous Local<br/>Kernel Evaluation"]
    Eval --> Control["Geodesic Optimization<br/>and Density Control"]
    Control --> Render["Intersection-Based<br/>Texture Rendering"]
    Supervision["UV or multi-view supervision"] -.->|Training error| Control
    Render -.->|Training feedback| Control
    Render --> Output["Albedo and rendered images"]

Key Designs

1. Spectral Precomputation and Alignment: remove per-kernel eigendecomposition from the optimization loop

Heat kernels require the eigenvalues and eigenfunctions of the Laplace-Beltrami operator (LBO), while anisotropic kernels additionally depend on diffusion orientation and anisotropy strength. Rebuilding and decomposing an anisotropic LBO (ALBO) whenever any kernel changes shape would make continuous optimization prohibitively expensive. The authors precompute a discrete grid over angle and anisotropy, using \(s=\ln(1+\eta)\) as the anisotropy coordinate so that highly anisotropic regions do not dominate the sampling scale. For a continuous parameter query, they locate the surrounding grid cell and bilinearly interpolate the eigenvalues and eigenvectors at its four corners. The implementation uses \(7\times7\) parameter combinations with 256-dimensional ALBO spectral information per combination, plus a separate 64-dimensional isotropic spectrum for distance queries.

Matrices from independent decompositions cannot simply be interpolated because neighboring parameters may produce eigenvector permutations, sign flips, and subspace rotations. The authors first match permutations using a mass-weighted correlation matrix and the Hungarian algorithm, align rotations through orthogonal Procrustes, and finally correct signs. The first reference comes from the isotropic LBO; subsequent alignment follows grid construction using an already aligned basis, and eigenvalues are reordered with the permutation. These operations make the spectral coordinates connected by interpolation more consistent instead of mechanically blending columns with different meanings. Parameter interpolation nevertheless remains a computational approximation, not an exact eigendecomposition of a new ALBO at every query.

2. Continuous Local Kernel Evaluation: decouple texture detail from vertex resolution

Each kernel stores a surface center, orientation, anisotropy strength, scale threshold, boundary sharpness, and RGB color. The center is represented by a triangle index and barycentric coordinates, so it can lie inside a face rather than being restricted to a vertex. Target queries use the same surface localization; spectral quantities precomputed at vertices are barycentrically interpolated at both the source and target. The heat response combines eigenfunction values at these two locations with diffusion weights that decay according to the eigenvalues. This is not vertex-color interpolation: the interpolated quantities are geometric spectral bases, while final color also depends on learnable local kernel composition.

Changing diffusion time affects both kernel width and source temperature, coupling scale and color optimization. The authors divide the heat response by its post-diffusion value at the center, normalizing the center to 1, and keep diffusion time fixed. The normalization in the paper's Eq. (4) can be written as:

\[ \bar h_t(p\mid p^\star,\theta,\eta) =\frac{h_t(p,p^\star\mid\theta,\eta)}{h_t(p^\star,p^\star\mid\theta,\eta)}. \]

Finite spectral truncation can introduce Gibbs-like ringing far from the center, so not every nonzero response should contribute meaningful color. The authors suppress distant responses with Gaussian decay based on biharmonic distance, then apply a rescaled sigmoid soft step to control extent and boundary sharpness. The scale threshold controls how much of the region remains, while sharpness determines whether it resembles a soft color patch or a crisp geodesic ellipse. These operations allow a smooth spectral response at fixed diffusion time to represent sharper and more localized texture boundaries.

To avoid traversing all kernels for every query, the authors scale isotropic eigenfunctions by their corresponding eigenvalues to construct a biharmonic spectral embedding. Euclidean KNN in this embedding retrieves nearby kernels according to approximate surface biharmonic distance rather than straight-line distance in three-dimensional space. They first select \(K_s\) candidate sources and evaluate anisotropic responses only for them, reducing local kernel evaluation from \(O(P\times S)\) to \(O(P\times K_s)\). This complexity statement concerns kernel evaluation; it does not imply that building and searching the KNN index is free. Among candidates, the \(K_c\) strongest responses are selected for color formation; \(K_s\) is the neighborhood budget, whereas \(K_c\) is the final blending budget. Selected colors are mixed using normalized responses and added as a residual to a global base color; regions without effective contributions revert to the base color rather than becoming transparent. For source mass, the paper proposes a density-estimation approach but uses unit mass \(m^\star=1\) in practice, reporting faster evaluation without degraded performance; the cache provides no numerical comparison table for this claim.

3. Geodesic Optimization and Density Control: keep movement and growth on the surface

Adding a three-dimensional gradient directly to a kernel center would move it off the mesh, so the authors first project the position gradient onto the current tangent plane. The projected vector defines an initial velocity, and an exponential map updates the position along a surface geodesic instead of moving off-surface and projecting back to a nearest point. A straightest-geodesic algorithm handles triangle crossings, while accumulated optimizer momentum is parallel-transported into the tangent space at the new position. This keeps historical update directions compatible with the current surface orientation; momentum from the old tangent plane cannot simply be reused unchanged. Positions are updated with GPU Riemannian SGD, while other kernel attributes use Adam and projected-gradient steps enforce parameter ranges.

To remove ineffective kernels, the system accumulates hit counts and energy over a window: hits count selection in the final blending set, while energy sums filtered responses. Kernels are pruned when their hit count is too low relative to the maximum, or when accumulated energy falls below a threshold. Adding detail instead depends on local reconstruction error: pointwise \(L_1\) color errors are distributed to contributing kernels using normalized blending weights, yielding a per-kernel error score. A high-error kernel with small extent is cloned in place so that later optimization can separate the copies; a large one is split into two smaller kernels. Split positions are displaced along the principal axis through the exponential map, keeping new kernels on the surface as well. A cap on the growth ratio prevents a single error spike from causing uncontrolled primitive proliferation. The \(L_1\) error used for density control should not be confused with the squared-error objective used to fit UV colors.

4. Intersection-Based Texture Rendering: query surface color instead of projecting kernels onto the screen

During rendering, a ray first intersects the original triangle mesh; that intersection triggers a local heat-kernel query whose color becomes the point's albedo. Kernels are not projected into screen-space ellipses, nor are they depth-sorted for volumetric opacity compositing. Thus, "Do Not Splat" describes the evaluation and image-formation mechanism, not the absence of Gaussian-like local support. The implementation wraps PyTorch evaluation through Dr.Jit to expose HKTex as a differentiable texture in Mitsuba3. Pixel losses back-propagate through ray tracing into texture parameters, allowing the same representation to support direct color fitting and inverse rendering. At test time, fixed parameters allow the KNN database and center self-responses to be prepared once; changing source positions during training requires corresponding query-state updates. Avoiding repeated preparation reduces overhead, but the reported tables still show substantially slower evaluation than conventional texture lookup.

A Worked Example

Consider a known mesh with a narrow color boundary on a curved surface; this is an explanatory example, not an additional experiment. The system computes spectral bases for the mesh, initializes kernel centers on the surface, and reads supervision colors from the UV texture at sampled points. A sample first retrieves candidate kernels in the biharmonic embedding, evaluates their responses through interpolated spectral bases, and blends the strongest contributions. If the boundary is too blurry, optimization can change orientation, threshold, and sharpness without subdividing the entire geometric mesh for color detail. If the region still has high error, density control clones small kernels or splits large ones to allocate more texture degrees of freedom there. Kernels remain on the surface when crossing triangles, and intersections observed from different cameras subsequently share the same continuous texture.

Loss & Training

With an existing UV texture, predicted and ground-truth colors are compared at matching surface samples; Section 4.4 uses squared error:

\[ \mathcal L=\lVert C(p)-C_{\mathrm{GT}}(p)\rVert_2^2. \]

The task description introduces uniform face sampling, whereas the implementation states that importance sampling is used; its details are deferred to supplementary material absent from the cache. The multi-view setup assumes a given mesh and cameras, samples minibatches from rays hitting the mesh across random views, and updates albedo through rendering error. Training uses a path-replay back-propagation (PRB) integrator with ray depth 3, rather than supervision that only compares direct surface colors. Position initialization combines area-based oversampling with farthest-point sampling; colors are initialized from \([-1,1]\), with their mean moved into the global base color. Initial orientations span \([0,\pi]\), anisotropy strengths span \([1,100]\), and boundary sharpness is initialized to 10–50. Threshold initialization spans 0.9–1.0 for UV fitting and 0.7–1.0 for the multi-view setup. The authors tune hyperparameters with Optuna on 5 meshes excluded from evaluation, but the cache does not provide the full learning-rate, neighbor-count, and controller-threshold configuration.

Key Experimental Results

Main Results

The data come from filtered Objaverse assets with labels and valid textures, a maximum of 60,000 vertices, and exclusions including non-manifold meshes, disconnected components, and failed eigendecompositions. Approximately 3,000 models remain after filtering; UV-fitting evaluation uses 313 of them, while multi-view fitting uses 162. Image metrics are calculated from 5 rendered views per object; the tables retain the reported means and standard deviations rather than treating scaled values as raw metrics.

Table A excerpts the paper's Table 1 on page 14, evaluating existing UV albedo fitting; LPIPS is reported in units of \(10^{-2}\), SSIM in units of \(10^{-2}\), and higher PSNR is better.

Method	PSNR ↑	LPIPS ↓ (×10^-2)	SSIM ↑ (×10^-2)	Render time (ms) ↓	Storage (KB) ↓
LR GT UVTextures	48.4 ± 15.2	2.3 ± 4.2	98.2 ± 3.4	33.7 ± 3.2	115.8 ± 103.2
GT VTex	37.9 ± 8.5	4.0 ± 5.0	96.8 ± 4.2	25.4 ± 3.1	17.3 ± 32.6
HR GT VTex	45.3 ± 5.7	1.1 ± 1.7	98.8 ± 2.1	50.3 ± 49.4	179.6 ± 89.5
MLP Pos. Enc.	42.9 ± 6.6	2.3 ± 2.5	98.2 ± 2.2	90.9 ± 35.2	551.3 ± 759.6
HKTex	44.8 ± 5.8	1.3 ± 1.6	98.9 ± 1.5	784.7 ± 474.3	96.6 ± 12.2

HKTex uses approximately 4.8k kernels on average in this setting and has strong SSIM, but its PSNR is below LR GT UVTextures and its LPIPS is worse than HR GT VTex. At the displayed precision, its Table 1 MSE ties HR GT VTex at \(0.08\pm0.20\) (×10^-3), so it should not be described as uniquely best. The neural baselines have substantially larger actual storage footprints; although budget matching was a design goal, the final results are not a strictly equal-byte comparison across all methods.

Table B excerpts the paper's Table 2 on page 15 to analyze multi-view albedo recovery; NVDiffRec* fixes the ground-truth mesh and environment map and outputs only albedo, rather than using its full joint-reconstruction setting.

Method	PSNR ↑	LPIPS ↓ (×10^-2)	SSIM ↑ (×10^-2)	Render time (s) ↓	Storage (KB) ↓
MLP Pos. Enc.	35.75 ± 7.76	3.7 ± 3.8	97.2 ± 3.1	0.68 ± 0.12	602.71 ± 755.89
NVDiffRec*	36.40 ± 6.83	3.4 ± 3.6	97.4 ± 2.9	0.66 ± 0.11	516.12 ± 722.66
HR VTex	37.16 ± 5.69	3.1 ± 3.4	97.5 ± 2.8	0.58 ± 0.08	81.06 ± 131.16
HKTex	37.61 ± 4.71	2.1 ± 1.9	98.3 ± 1.6	1.2 ± 0.3	78.73 ± 23.12

This setup uses approximately 3.8k kernels on average; HKTex improves image quality and reported storage over the listed baselines, but rendering remains slower. Subtracting the means in Table B gives a PSNR advantage of 1.21 over NVDiffRec*, while LPIPS decreases from 0.034 to 0.021; these are within-table comparisons, not additional experiments.

Ablation Study

The available main-paper cache contains no numerical component-ablation table, so it cannot establish separate gains for spectral alignment, geodesic updates, or density control. Section 4.2 only qualitatively reports no degradation from unit mass; the two-triangle image-fitting example in Section 5.4 uses a pre-filtering exponent \(a=30\), while most meshes use \(a=1\). The latter is a configuration observation for a specific setting, not an ablation establishing a universal hyperparameter choice across meshes.

Key Findings

• Storage and speed show a clear trade-off: Table 1 reports 96.6 KB and 784.7 ms for HKTex versus 115.8 KB and 33.7 ms for LR GT UVTextures, so storage alone is an incomplete comparison.
• Multi-view results support appearance recovery with fixed geometry, not a direct claim about jointly estimating geometry, illumination, and materials from real photographs.
• The authors apply identical zip compression and count baseline topological overhead such as UV coordinates, but do not explicitly itemize whether HKTex's precomputed spectral bases enter Storage; the reported footprint is not equivalent to runtime GPU memory.

Highlights & Insights

• Surface adherence becomes a shared constraint of representation and optimization rather than merely a regularizer. Kernel definition, position updates, and splitting all follow the surface instead of making inconsistent spatial assumptions.
• Geometric sampling and texture detail are separated. Spectral functions originate from the mesh, but learnable kernels can occupy face interiors and change boundaries, so color resolution is not directly determined by vertex count.
• Spectral alignment is a prerequisite for continuous parameter interpolation, not negligible preprocessing. Sign and permutation freedoms in discrete eigendecomposition directly affect the coherence of downstream differentiable queries.

Limitations & Future Work

• The authors evaluate only albedo; BSDF attributes such as roughness and specular components remain future extensions, so integration into PBR does not mean full material recovery has been demonstrated.
• Strong geometric filtering limits the evidence for complex interiors, multipart assets, and non-manifold inputs; being UV-free does not remove dependence on mesh quality.
• Rendering queries are expensive, and spectral precomputation time, working memory, and persistent storage are not fully separated in the main paper. Real-time deployment should assess them alongside the final kernel parameters.
• Text extraction damages several operator, filtering, and color-formation equations; this note retains only unambiguous mathematical relationships and explains other mechanisms from the prose. Exact implementation requires checking the typeset paper and supplementary material.
• Reader interpretation: deployment research should first examine shared spectral caches, neighborhood queries, and kernel aggregation rather than assuming fewer KB imply lower total resource costs.

Related Work & Insights

• Compared with 3DGS / SuGaR / MeshGS: these approaches use Euclidean Gaussians or additional surface-alignment constraints; HKTex textures a known mesh with heat kernels and does not address the same scope of scene reconstruction.
• Compared with Intrinsic Neural Fields: both exploit surface spectra, but the former conditions an MLP on interpolated spectral coordinates, while HKTex uses them for explicit local heat kernels with adaptive density.
• Compared with ImageGS / UV textures: two-dimensional representations still require a surface-to-atlas mapping, which HKTex avoids; complex surface queries replace mature two-dimensional texture lookups.
• Research direction: fixed-geometry material compression could test whether the same intrinsic primitives work across several material channels; this is a future direction, not an established result of the paper.

Rating

• Novelty: 4/5. Combines anisotropic heat kernels, surface optimization, and differentiable PBR into a texture representation distinct from merely aligning Euclidean Gaussians to a surface.
• Experimental Thoroughness: 3/5. Covers two tasks and multiple baselines, but the available main paper lacks numerical component ablations and leaves dataset and storage-accounting limitations.
• Writing Quality: 4/5. The problem and components connect clearly, although budget-matching statements require comparison with actual tables and several equations are damaged in text extraction.
• Value: 4/5. Useful for compact surface appearance and inverse-rendering research, but not yet a direct replacement for real-time UV texture systems.