3D Gaussian Texture for Real-time Mesoscale Appearance Synthesis and Rendering¶

Conference: ECCV 2026
Paper: ECCV official page · PDF
Code: https://github.com/ChenXiang0810/multilayer-gaussian-texture
Area: 3D Vision
Keywords: multi-layer Gaussian textures, mesoscale appearance, structure-preserving synthesis, deformation-aware mapping, material-lighting decomposition

TL;DR¶

The paper organizes 3D Gaussians into regular texture layers at similar heights above a surface, enabling real-time exemplar-based synthesis, transfer to new meshes, and basic relighting of structures such as grass, durian spikes, and wicker gaps; it improves SIFID over NeRF-Texture in all four comparisons, with reported times of 3.28–4.98 ms in Figure 4.

Background & Motivation¶

Mesoscale appearance lies between the overall shape of an object and a microscopic reflectance model. Grass blades and flowers in a lawn, or strands and gaps in wicker, contribute not just color but thickness, occlusion, parallax, and changing silhouettes. A conventional color texture cannot reproduce these effects, while explicitly modeling every element or using high-resolution volumes requires substantial authoring and storage. NeRF-Tex and NeRF-Texture learn such appearance from multi-view images: the former repeatedly places appearance primitives, while the latter also synthesizes larger, nontrivially repeating textures. However, their implicit representations remain limited in detail and runtime performance.

Replacing NeRF with fast, explicit 3D Gaussians is not enough. A conventional Gaussian collection is tied to its original scene and lacks the regular indexing needed for texture synthesis. Projecting it onto a single texture can place several Gaussians in the same texel and blend structures at different heights, such as a table and the fruit above it. Even after expanding the texture, transfer to a curved mesh can produce needle-like artifacts, and illumination baked into the original colors is unsuitable for a new scene.

The central problem is therefore to create a Gaussian representation that remains usable through training, synthesis, and transfer, rather than merely to replace the renderer. Core idea: organize Gaussians into regular height-separated texture layers, synthesize them while preserving spatial structure and material correlations, and use constrained mesh mapping plus material-lighting decomposition to turn local appearance into a reusable 3D texture.

Method¶

Overall Architecture¶

The input is a set of multi-view images of an object with mesoscale appearance. Training extracts a base mesh and learns a multi-layer Gaussian texture on it. At application time, the system synthesizes texture samples for the requested region, maps the resulting Gaussians to a new UV-parameterized mesh, decodes material properties, and renders with the chosen illumination. The learned asset is thus a transferable, extensible appearance representation and a material decoder, not just a fixed scene supporting novel views.

The diagram follows the application-time data flow. Material latents and geometric parameters are learned during training rather than predicted from scratch at deployment. During optimization, mapping, decoding, and rendering also connect back to image supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view images<br/>and extracted base mesh"] --> B["Multi-layer Gaussian texture"]
    B --> C["Structure-preserving synthesis"]
    C --> D["Deformation-aware mapping"]
    T["Target mesh and UVs"] --> D
    D --> E["Latent materials and<br/>lighting decomposition"]
    L["View and illumination"] --> E
    E --> F["Gaussian blending<br/>Appearance on a new object"]

Key Designs¶

1. Multi-layer Gaussian texture: make within-layer blending operate at similar heights

Instead of fitting an arbitrary Gaussian cloud and then forcing it into a 2D image, the method constructs a regular multi-layer texture with exactly one Gaussian per texel in each layer. Its shape is \(N\times H\times W\times D\), denoting the number of layers, the height and width of each layer, and the parameter count per Gaussian. Each Gaussian stores a 3D position, opacity, 3D scale, quaternion rotation, and an 8-dimensional material latent; shading normals are also temporarily stored during training. Learnable opacity allows cells without meaningful structure to remain visually inactive rather than requiring every cell to be opaque.

The crucial restriction keeps each Gaussian near its corresponding texel center in a small 3D region. Texture resolution determines the horizontal extent, while estimated structure thickness and layer count determine its vertical extent. Gaussians in the same layer consequently have similar heights above the surface, so layer-wise synthesis is less likely to mix the base surface with elevated protrusions. The layers do not merely add color channels: they preserve vertical organization in a regular index structure, giving 2D texture operations meaningful 3D elements to manipulate.

2. Structure-preserving synthesis: handle positions, opacity, and pattern correlations together

For a target query, exemplar-based synthesis selects several source patches and blends their attributes using weights. Directly averaging absolute Gaussian positions would pull the result toward the source patches rather than place it around the target query. The paper therefore uses the intact relation \(d=\mu-p\) to express each position as an offset from its texel center, blends these offsets, and adds the target query position back. This separates a sample's texture address from the local arrangement of geometry inside its cell, allowing the same detail to appear at a new location.

Opacity cannot be averaged indiscriminately either: mixing opaque structure with transparent space creates semitransparent geometry absent from the exemplar. The method divides samples using \(o_m=0.1\) and sums the original weights assigned to samples at or above this threshold. If that mass reaches 0.5, only that group is retained; otherwise, only the below-threshold group is retained. The surviving weights are renormalized. All layers use source patches with matching positions and sizes so that independent sampling does not tear apart cross-layer structure. On top of these adaptations, autocovariance computed from opacity and latent features guides patch selection for texture translations, while variance-preserving blending reduces contraction of contrast and color statistics. Autocovariance captures correlations at different spatial offsets, so selection considers the arrangement of patterns such as strands and brick joints, not just their average color.

3. Deformation-aware mapping: transport Gaussians with UV barycentric coordinates and constrain problematic scales

The synthesized texture still lives in canonical coordinates and must be transferred to a target object. For a UV-parameterized mesh, the method identifies the triangle containing the projection of each Gaussian center onto the UV plane and computes its barycentric coordinates. Corresponding 3D vertices determine its surface location. Height above the texture plane is transported along the target face normal, accounting for the scale change from the UV triangle to the mesh triangle; Gaussian orientation and scale are updated as well. Direction-dependent attributes, including normals and spherical harmonics, must rotate consistently rather than remaining in their original frame.

The scale constraint addresses artifacts before rendering. During training, it limits Gaussian scales according to texel size, reducing excessively elongated or oversized primitives. Such Gaussians may fit images on the original object but turn into visible needles after transfer to a curved surface. The representation is therefore regularized for the mapping operation it must later support. This does not eliminate dependence on base-mesh quality or UV parameterization, and it should not be interpreted as solving arbitrary complex geometry.

4. Latent materials and lighting decomposition: preserve correlations between synthesized material attributes

Standard 3DGS color spherical harmonics encode the illumination present during capture, making them unsuitable for direct relighting. The method instead uses illumination spherical harmonics and a Phong material model. A small MLP decodes an 8-dimensional latent into 3D diffuse color, scalar specular intensity, scalar glossiness, and a 3D shading normal. Given a view direction, spherical-harmonic convolution produces the Gaussian color used in pixel blending. Synthesizing latent features rather than independently blending every material channel is intended to preserve correlations between color, highlights, and normals rather than generate mutually inconsistent attributes.

An interpolation-friendly latent space is not guaranteed by unconstrained per-Gaussian optimization. The intermediate training stage therefore introduces an encoder that takes previously trained color spherical harmonics and shading normals and produces material latents, regularizing the representation together with the decoder. The final stage removes the encoder and directly optimizes the latents and decoder. Deployment retains only the small decoder, while the training-time constraint improves synthesis behavior. This supports basic relighting, not complete global light transport or unrestricted complex illumination.

A Worked Example¶

The fruit-on-a-table ablation illustrates how these operations interact. After training, lower texture layers mainly represent the table and higher layers represent the fruit. To extend the appearance, each layer uses the same source patch positions and sizes but blends only its own Gaussians. Fruit colors therefore do not get mixed into the table merely because their samples are adjacent in a 2D projection. Center-relative offsets place synthesized Gaussians around the new queries, and opacity grouping avoids mixing solid fruit with nearby empty space into extra semitransparent colored patches.

When the resulting appearance is transferred to a curved target mesh, UV barycentric coordinates determine placement, the scale constraint reduces stretching artifacts, and decoded latent materials respond to the new lighting. The first part describes the actual Figure 7 ablation scene; the transfer continuation explains how the paper's operations compose and is not an additional quantitative experiment.

Loss & Training¶

Training has three stages. The first learns non-material parameters using color spherical harmonics, avoiding simultaneous material decomposition and initial geometry fitting. Shading normals receive supervision from pseudo normals computed from depth. The second adds illumination spherical harmonics and encoder-decoder material learning. The third discards the encoder and continues optimizing the material latents and decoder.

The first stage uses an L1 image-color loss, a position restriction around each texel, the scale constraint, a depth-distribution loss that draws Gaussians toward the object surface, and normal consistency. The later stages retain color and geometry supervision, supervise decoded normals, and add smoothness constraints on diffuse color and specular intensity. The weights explicitly stated in the text are \(\lambda_d=0.01\), \(\lambda_n=0.1\), and \(\lambda_{k_d}=\lambda_{k_s}=0.01\).

The cached text has damaged piecewise formatting in Equation (1), corrupted mapping expressions in Equations (2)–(3), and missing operators or formatting in the constraints and objectives of Equations (4)–(9). This note retains only the confirmed offset relation, dimensions, and weights, explains the remaining mechanisms from adjacent prose, and does not reconstruct damaged formulas. Mesh extraction details, specific training settings, and additional relighting results are deferred to supplementary material that is absent from this cache.

Key Experimental Results¶

Main Results¶

Experiments use an NVIDIA RTX 4090. SIFID and memory below come from Table 1, while timings come from Figure 4. SIFID is the single-image version of Fréchet Inception Distance: it measures similarity between image-feature distributions, with lower values indicating greater similarity, rather than measuring geometric reconstruction error.

Example	Ours SIFID	NeRF-Texture SIFID	Ours memory	NeRF-Texture memory	Ours time	NeRF-Texture time
Grass, synthetic	10.86	15.94	280 MB	1.1 GB	3.41 ms	1.43 s
Durian, real	4.82	10.71	140 MB	1.1 GB	3.70 ms	1.24 s
Wicker, synthetic	4.38	6.89	140 MB	1.1 GB	4.98 ms	3.03 s
Flower, real	5.26	6.96	420 MB	855.7 MB	3.28 ms	1.67 s

Both SIFID and memory improve in all four examples. The discussion of Figure 4 emphasizes preservation of small yellow flowers in grass, durian spikes, wicker gaps, and flower shapes, not merely color fitting. Different learned illumination can also produce color differences, so those differences should not all be attributed to structural errors.

There is an unresolved timing discrepancy: Figure 1 reports about 20 microseconds, and Section 4.1 also says microseconds, whereas Figure 4 explicitly labels the proposed method's times in milliseconds. The table uses only the directly paired per-example Figure 4 numbers. The authors' approximate 500-fold acceleration claim should not become a fixed multiplier for every scene, and the cache does not establish that the microsecond figure measures a complete frame.

Ablation Study¶

The following SIFID results belong to different figures and scenes and should only be compared within each group. In particular, Figure 8's 0.87 is not a further improvement over the full configuration in Figure 7.

Evidence and configuration	SIFID, lower is better	Change from full method in the same group	Interpretation
Figure 7: full blending	1.43	Reference	Both multi-layer texture and opacity grouping
Figure 7: without multiple layers	1.69	+0.26	Different heights mix, transferring fruit color into the table
Figure 7: without opacity-threshold grouping	2.27	+0.84	Semitransparent blends introduce colored artifacts
Figure 8: random synthesis	1.81	+0.94	Brick structure is disrupted
Figure 8: autocovariance preservation only	1.35	+0.48	Structure improves, but color statistics still differ
Figure 8: autocovariance and variance preservation	0.87	Reference	Preserves structure and more exemplar-like color

Figure 7 also compares the scale constraint and the training encoder. Removing the former produces needle-like artifacts; removing the latter produces crossing artifacts. The cache does not provide clearly paired quantitative values for those two tests, so no SIFID numbers are invented for them. Direct repetition in Figure 8 visibly repeats patterns but also lacks a readable corresponding score.

Key Findings¶

Within Figure 7, removing opacity grouping increases SIFID more than removing multiple layers. This supports a local comparison for that example, not a universal ranking of module importance across materials.
Figure 8 shows that autocovariance and variance preservation are complementary: the former constrains pattern arrangement, while the latter reduces blending-induced changes in color statistics. Neither alone matches the combination.
Fast application does not imply cheap training. Gaussian positions change during optimization, requiring repeated UV barycentric-coordinate computation; the authors report training times of 0.5–3 hours.

Highlights & Insights¶

The reusable contribution is the joint design of representation and operations. One Gaussian per texel, height separation, local offsets, and opacity grouping jointly specify which primitives can be meaningfully blended rather than simply applying a 2D algorithm to a point cloud.
Empty space is treated as part of structure. In grass and wicker, the arrangement of transparent regions matters alongside solid elements, explaining why a seemingly simple opacity rule can strongly affect synthesis quality.
The encoder supplies latent-space regularization only during training. This strategy may be useful for editable appearance models requiring both fast deployment and stable attribute interpolation, although transfer to other tasks would need validation.

Limitations & Future Work¶

The authors show failure on complex geometry without clear mesoscale structure. The method relies on mesh extraction and manageable UV coordinates and is not a general-purpose system for reconstructing complex object geometry.
Training takes 0.5–3 hours, depending on texture resolution and scene complexity. More efficient coordinate updates or parameterizations are plausible research directions, not demonstrated improvements in this paper.
Spherical-harmonic illumination and Phong materials support only basic relighting. Complex illumination and complete relighting remain future work; Figure 5's qualitative examples do not establish physical accuracy.
From an evaluation standpoint, the main table contains only four examples and one principal baseline. SIFID and memory alone cannot establish accurate 3D structure or cross-view consistency; broader geometry, viewpoint, material, and illumination tests would strengthen the evidence.
The available text lacks supplementary material and contains damaged formulas and conflicting timing units. It is insufficient for reproducing all implementation details or resolving the microsecond-versus-millisecond measurement scope.

NeRF-Tex / NeRF-Texture: the former repeatedly places neural appearance primitives, while the latter supports nonrepetitive synthesis. The proposed explicit Gaussian texture improves runtime and detail but must directly address position blending, opacity, and deformation.
UV Gaussians / Gaussian Shell Maps: these also organize Gaussians through textures or shell-like representations, mainly for human representation and deformation. This paper additionally requires synthesis of new texture regions and transfer of learned appearance to new objects.
Quilting and exemplar-based synthesis: sequential patch matching or optimization typically requires generating a substantial texture before use. Independent exemplar-based queries are better suited to parallel, on-the-fly generation. The paper reuses statistical-preservation ideas and contributes a Gaussian representation compatible with them.
Relightable 3D Gaussians: both approaches involve material-lighting decomposition, but this work also needs material attributes to interpolate meaningfully during synthesis, motivating an encoder-regularized latent material space.

Rating¶

Novelty: 4/5. Height-separated geometry, synthesizable Gaussian parameters, and transferable materials form a coherent representation; several synthesis and relighting components build on prior work.
Experimental Thoroughness: 3/5. Real and synthetic data, runtime and memory comparisons, and several ablations provide evidence, but the main table is small, relighting is mostly qualitative, and timing scopes conflict.
Writing Quality: 4/5. Representation, synthesis, mapping, and training form a clear argument, and the figures explain failure mechanisms; corruption in the cached formulas is not attributed to the original paper.
Value: 4/5. Practically relevant for real-time graphics requiring extensible mesoscale detail, provided offline training, base-mesh dependence, and limited relighting are acceptable.