Geometry-Aware Style Transfer in 3D Gaussian Splatting¶

Conference: ECCV 2026
arXiv: 2606.24144
Code: https://github.com/oweixx/gast
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Style Transfer, Decoupled Optimization, Contrastive Learning, Geometry-aware

TL;DR¶

This paper proposes a geometry-aware 3DGS style transfer framework. By alternately updating color and geometry parameters via decoupled optimization and introducing a multimodal (RGB + depth + edge) contrastive feature matching loss (GCFM) to guide the transfer of style structure and geometric texture, it significantly outperforms existing 3DGS style transfer methods in terms of style fidelity and multi-view consistency.

Background & Motivation¶

2D image style transfer is highly mature, with successful solutions ranging from Gatys' Gram matrix to diffusion models, yet it cannot guarantee cross-view consistency. With the rise of NeRF and 3DGS, 3D style transfer has become possible—among which 3DGS, relying on its explicit geometric primitives and real-time rendering capabilities, has become the dominant backbone in this field.

However, existing 3DGS style transfer methods generally adopt a conservative strategy: only transferring color/appearance attributes while enforcing strong constraints to keep geometry parameters unchanged. The reason is that 3DGS rendering results are highly sensitive to changes in geometric parameters—color modifications only affect the surface appearance, whereas directly modifying geometric parameters (position, covariance, opacity) alters the underlying structure of the scene, and even minor geometric miscalculations can severely damage visual quality. This is analogous to pasting wood-grain wallpaper on the surface of furniture: the paper can conform to the shape, but can never change the structure to reflect the bump textures and structural irregularities of real wood.

This forms a Core Problem (Key Challenge): geometry is an important carrier of style (such as the thickness of Van Gogh's brushstrokes or the blocky structures of Cubism), but directly optimizing 3DGS geometric parameters easily leads to scene collapse. Existing methods either only optimize color (StyleGaussian, SGSST), only perform limited geometric adjustments in the pipeline pre-processing stage (Gaussian splitting in GStyle), or constrain the geometry to remain unchanged using depth-preservation losses (StylizedGS, ABCGS)—essentially avoiding geometric stylization rather than treating geometry as a medium for style expression.

Core Idea: Color and geometry are treated as two equal dimensions of stylization. Through decoupled alternating optimization, update interference between the two is eliminated, and multimodal contrastive feature matching (simultaneously considering RGB, depth, and edges) is utilized to guide the alignment of geometric structures with the target style.

Method¶

Overall Architecture¶

The goal of this paper is to transfer the artistic style of a 2D style image \(\mathbf{I}^{\mathcal{S}}\) to a 3DGS scene reconstructed from a set of multi-view content images \(\mathbb{I}^{\mathcal{C}}\) while maintaining spatial and geometric consistency. The overall pipeline consists of three steps:

First, an initial Gaussian representation \(\mathbb{G}^{\text{init}}\) is reconstructed from the content images using a standard 3DGS pipeline. Second, a linear transformation is applied to the Gaussian colors using the color matching from ARF to align the initial color distribution with the target style, yielding \(\mathbb{G}^{\text{c}}\). The third step is the core—transitioning through decoupled optimization to alternately execute the color optimization stage and geometric optimization stage in an outer loop, both of which are driven by the proposed GCFM loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Content Images"] --> B["3DGS Initial Reconstruction"]
    B --> C["Color Matching<br/>Linear transformation to align style color distributions"]
    C --> D["Color Optimization Stage<br/>Freeze geometry, update DC color"]
    D --> E["Geometry Optimization Stage<br/>Freeze color, update position/covariance/opacity"]
    E -->|K rounds of outer loop| D
    E --> F["Stylized 3DGS Scene"]
    G["Style Image"] --> H["Multimodal Feature Extraction<br/>RGB + Depth + Edge"]
    H --> I["GCFM Contrastive Loss<br/>Pull positive samples/Push negative samples"]
    I -.->|Drives both stages| D
    I -.->|Drives both stages| E

Key Designs¶

1. Decoupled Optimization: Alternately updating color and geometry to eliminate mutual interference of joint optimization

Existing methods either optimize color while freezing geometry, or jointly optimize both but face severe optimization conflicts. During joint-step optimization, the gradients of color and geometry propagate simultaneously, easily leading to color over-stylization—the optimization process erroneously modifies geometry to embed low-level style texture patterns, which in turn destroys the scene structure (e.g., the flower structure in Fig. 3 gradually collapsing across iterations).

The decoupled optimization proposed in this paper divides the entire optimization process into \(K\) outer loops, solving two sub-problems sequentially in each round:

\[\boldsymbol{\Theta}^{k+1} = \arg\min_{\boldsymbol{\Theta}} \mathcal{L}(\boldsymbol{\Theta}, \boldsymbol{\Phi}^k) \quad \text{(Color Stage)}\]

\[\boldsymbol{\Phi}^{k+1} = \arg\min_{\boldsymbol{\Phi}} \mathcal{L}(\boldsymbol{\Theta}^{k+1}, \boldsymbol{\Phi}) \quad \text{(Geometry Stage)}\]

where \(\boldsymbol{\Theta}\) denotes the color parameters (only optimizing the zero-order DC component of the spherical harmonics coefficients, i.e., basic RGB, while leaving higher-order coefficients frozen), and \(\boldsymbol{\Phi} = \{\boldsymbol{\mu}_n, \boldsymbol{\Sigma}_n, \alpha_n\}\) denotes the geometric parameters (positions, covariances, opacities). In practice, each sub-problem is approximately solved using a fixed number of gradient descent steps: \(N_c = 10\) steps for the color stage, \(N_g = 90\) steps for the geometry stage, and \(K = 30\) rounds of outer loops, totaling 3,000 iterations.

Why it works: The essence of alternating optimization is establishing a "mutual guidance loop"—first updating color brings the appearance close to the target style, providing a stable visual reference for subsequent geometric updates; then, when updating the geometry, having a reliable color foundation allows for more accurate judgments of which structural changes are required by the style rather than being noise. Ablation studies also confirm that the SIFID of joint optimization (1.4220) is significantly worse than that of decoupled optimization (1.1736), accompanied by scene collapse due to over-stylization of color.

2. GCFM: Multimodal Contrastive Feature Matching, Jointly Guiding Geometric Style Alignment with RGB + Depth + Edge

Traditional feature matching in 3D style transfer (such as in ARF) typically performs nearest-neighbor matching only in the RGB domain, lacking awareness of geometric structures. The proposed GCFM captures style features from three dimensions:

Color Features \(\mathbf{F}_{\text{color}}\): Standard RGB features extracted from VGG, capturing texture and color style.
Depth Features \(\mathbf{F}_{\text{depth}}\): Depths of rendered views are directly obtained from the 3DGS rendering pipeline, while depths of style images are estimated by DepthAnythingV2. Depth provides coarse-grained spatial structural information.
Edge Features \(\mathbf{F}_{\text{edge}}\): Extracted using the Canny operator on depth maps rather than color maps (as style images possess complex textures and noisy color edges). Edges capture structural boundaries to guide contour-level shape alignment.

The tri-modal features are concatenated channel-wise to form a joint feature map \(\mathbf{F}^s = [\mathbf{F}_{\text{color}}^s, \mathbf{F}_{\text{depth}}^s, \mathbf{F}_{\text{edge}}^s]\) (where \(s \in \{\mathcal{R}, \mathcal{S}\}\) represents the rendered view and the style image, respectively).

Unlike previous works that only use positive sample (nearest neighbor) matching, GCFM introduces a contrastive learning mechanism: for each spatial position \(\mathbf{x}\) in the rendered feature map, it identifies the most similar positive sample \(\mathbf{y}^+(\mathbf{x})\) and the least similar negative sample \(\mathbf{y}^-(\mathbf{x})\) in the style feature map:

\[\mathbf{y}^+(\mathbf{x}) = \arg\min_{\mathbf{y}} D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}))\]

\[\mathbf{y}^-(\mathbf{x}) = \arg\max_{\mathbf{y}} D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}))\]

where \(D_{\cos}(\cdot, \cdot) = 1 - \cos(\cdot, \cdot)\) is the cosine distance. The GCFM loss pulls rendering features (anchors) toward positive samples and pushes them away from negative samples:

\[\mathcal{L}_{\text{GC}} = \frac{1}{|\Omega_{\mathbf{F}}|} \sum_{\mathbf{x} \in \Omega_{\mathbf{F}}} \Big(2 + D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}^+(\mathbf{x}))) - D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}^-(\mathbf{x})))\Big)\]

Ablation results indicate that removing the contrastive objective (i.e., only positive matching) leads to blurred structural features in depth maps; using a single modality (RGB only) overly simplifies geometric contours. The full GCFM (tri-modal + contrastive) produces the sharpest geometric contours and the most faithful stylized structure replication.

Loss & Training¶

The total loss function is a weighted sum of four terms, with different combinations of terms utilized in the color and geometry stages:

\[\mathcal{L} = \lambda_{\text{GC}} \mathcal{L}_{\text{GC}} + \lambda_{\text{cont}} \mathcal{L}_{\text{cont}} + \lambda_{\text{TV}} \mathcal{L}_{\text{TV}} + \delta_{\text{reg}} \mathcal{L}_{\text{reg}}\]

GCFM Loss \(\mathcal{L}_{\text{GC}}\) (\(\lambda_{\text{GC}} = 2.0\)): The multimodal contrastive feature matching loss described above, serving as the core driving term of style transfer.
Content Preservation Loss \(\mathcal{L}_{\text{cont}}\) (\(\lambda_{\text{cont}} = 1 \times 10^{-3}\)): VGG-16 perceptual loss, which constrains the feature distance between rendered and content views to prevent scene structure destruction.
TV Regularization \(\mathcal{L}_{\text{TV}}\) (\(\lambda_{\text{TV}} = 0.02\)): Imposes local spatial smoothness constraints horizontally and vertically on rendered images, suppressing high-frequency noise and boundary artifacts.
Geometric Regularization \(\mathcal{L}_{\text{reg}}\) (only enabled in the geometry stage, \(\delta_{\text{reg}} = 1\)): Constrains optimized opacities \(\boldsymbol{\alpha}\), scales \(\mathbf{s}\), and rotations \(\mathbf{r}\) to not drift too far from their matched initial values, while additionally enforcing consistency between the rendered depth map \(\mathbf{D}^{\mathcal{R}}\) and reference depth map \(\mathbf{D}^{\text{c}}\), preventing Gaussian inflation, collapse, or global blurring.

The optimizer used is Adam with a learning rate of \(\eta = 1 \times 10^{-4}\) (reduced to \(2.5 \times 10^{-5}\) for the T&T dataset). Training is conducted on a single RTX A6000 (48GB), requiring approximately 14 minutes on average to stylize a single scene.

Key Experimental Results¶

Main Results¶

Across 8 scenes (4 from LLFF, 2 from T&T, and 2 from MipNeRF-360) and 9 style images, there are 72 scene-style combinations in total. SIFID measures style fidelity (lower is better), while short-term and long-term consistency are evaluated using optical flow warping + masked RMSE/LPIPS.

Method	SIFID ↓	Short-term LPIPS ↓	Short-term RMSE ↓	Long-term LPIPS ↓	Long-term RMSE ↓
StyleGaussian	3.3405	0.050	0.053	0.141	0.114
G-Style	1.2955	0.045	0.035	0.118	0.085
StylizedGS	2.2439	0.028	0.021	0.072	0.062
SGSST	1.3936	0.048	0.048	0.129	0.109
CLIPGaussian	3.3685	0.048	0.046	0.128	0.108
Ours	1.1736	0.042	0.034	0.114	0.084

Ours achieves the best SIFID (1.1736), significantly surpassing the second-best G-Style (1.2955). In terms of multi-view consistency, after excluding StylizedGS whose metrics are artificially inflated due to over-smoothing and desaturation, ours ranks first in both short-term and long-term LPIPS/RMSE. Concurrently, in a user study with 37 participants, ours received the highest scores across three dimensions: style similarity, visual appeal, and content legibility.

The average stylization time is 14 minutes and 14 seconds, which is significantly faster than StyleGaussian (410 minutes) and SGSST (59 minutes), achieving the best trade-off between speed and quality.

Ablation Study¶

Ablation on Optimization Strategies (Table 3): Contrasts the influences of four different strategies—optimizing geometry only, optimizing color only, joint-optimization (joint-step), and decoupled optimization—on SIFID and consistency.

Configuration	SIFID ↓	Short-term LPIPS ↓	Short-term RMSE ↓	Long-term LPIPS ↓	Long-term RMSE ↓
Geometry Only	2.5496	0.045	0.040	0.113	0.096
Color Only	1.5392	0.042	0.037	0.115	0.086
Joint Optimization	1.4220	0.042	0.036	0.122	0.087
Decoupled Optimization (Ours)	1.1736	0.042	0.034	0.114	0.084

Geometry Only achieves the worst SIFID (2.5496) because it lacks color adaptation to effectively match style distributions. Color Only improves the SIFID but is still limited (1.5392) as it cannot capture geometric style cues. Joint Optimization achieves an intermediate SIFID (1.4220) but performs worst in long-term consistency due to color over-stylization. Decoupled Optimization outperforms others on all metrics, proving that decoupling color and geometric updates stabilizes optimization and achieves a harmonious fusion of appearance and geometry.

GCFM Ablation: Removing the contrastive objective leads to blurred structural features in depth maps; using a single modality (RGB only) results in over-simplified geometric contours. The full GCFM yields the sharpest geometric contours and the most faithful structural replication. The SIFID stabilizes within 1.1683–1.1878 for GCFM weights \(\lambda_{\text{GC}}\) in the range of 1.0–10.0, demonstrating excellent robustness.

Ablation on Geometry/Color Update Ratios: Setting \(N_g = 90, N_c = 10\) achieves the optimal SIFID (1.1736); with \(N_g = 100\) (pure geometry), rigidity is too strong and lacks color adaptation, while \(N_g = 30\) leads to insufficient geometric stylization.

Key Findings¶

Decoupled optimization contributes the most: compared to joint optimization, the SIFID drops from 1.4220 to 1.1736, while simultaneously resolving issues of color over-stylization and scene collapse observed in joint optimization.
Geometric optimization is essential: the SIFID for color-only optimization is 1.5392, and it improves to 1.1736 when geometric optimization is incorporated, demonstrating that geometry is indeed a vital dimension for style representation.
The multimodal design of GCFM is critical: the complementary information of depth + edge enables structure-aware feature matching, whereas single RGB cannot capture geometric style.
Geometric updates need to dominate the optimization process (\(N_g : N_c = 90 : 10\)), which reflects that geometric stylization requires more fine-grained iterations than color stylization.

Highlights & Insights¶

Decoupling is Stability: Color and geometry optimization naturally exhibit gradient conflicts. Alternating updates serve as the simplest yet highly effective solution. This "divide and conquer" paradigm provides valuable reference for any 3D tasks involving the joint optimization of multiple parameter groups, without necessarily resorting to complex gradient modulation or adversarial training.
Multimodal Feature Concatenation for Contrastive Matching: Channel concatenation of RGB + depth + edge is simple and direct, but extracting edges from the depth map (while depths are obtained from the rendering pipeline) is a highly thoughtful design choice. It avoids the pitfall of noisy color edges caused by complex style image textures, reflecting meticulous consideration over "what information is extracted from which source."
Explicit Negative Samples in Contrastive Learning: Conventional feature matching only utilizes positive nearest neighbors. GCFM explicitly pushes away negative furthest neighbors, forcing the model to learn more discriminative style representations. This concept can be transferred to other tasks demanding cross-domain feature alignment (e.g., 3D editing, reference-based control in text-to-3D).
Geometric Regularization Anchoring the Initial State: Constraining parameters during the geometric optimization phase from drifting too far from their post-color-matched state is essentially a "trusted initialization" strategy—allowing the model to explore geometric deformation under guidance rather than freely mutating in an unconstrained manner, which would lead to collapse.

Limitations & Future Work¶

Limitations not explicitly discussed by the authors: The method relies on DepthAnythingV2 to estimate the depth map of the style image. For highly abstract/non-realistic styles (such as pure color blocks or minimalist lines), depth estimation can be inaccurate or even meaningless. Under such circumstances, the depth and edge branches of GCFM might introduce noise instead of useful signals.
Although stylization is faster than most previous methods, it still takes about 14 minutes per scene, which is far from real-time interaction. The number of Gaussians per scene (0.54M–4.20M) imposes high GPU VRAM requirements (up to 39 GB).
The "degree" of geometric stylization is controlled by the target texture scale and the \(N_g / N_c\) ratio, lacking more intuitive user-controllable parameters (such as a "geometric deformation intensity" slider), which is less user-friendly for artistic creators.
Future Directions: (1) Replace fixed concatenation with learnable depth/edge weights, allowing the model to adaptively determine the importance of different modalities under various styles; (2) Explore feed-forward geometric stylization networks to break free from the constraints of scene-specific optimization; (3) Incorporate prior knowledge from diffusion models to guide geometric deformation at a more abstract semantic level, rather than being confined to pixel-level feature matching.

vs ARF / SNeRF / CoARF (NeRF Style Transfer): These methods execute feature-matching style transfer on NeRF, but NeRF's implicit representation yields slow rendering and difficult geometric manipulation. This paper operates on the explicit primitives of 3DGS, naturally supporting real-time rendering and direct geometric modification.
vs StyleGaussian / SGSST: They only update color attributes while freezing geometry, simplifying style transfer to "reskinning 3D models." This paper proves that geometry can also serve as a style carrier, and decoupled optimization makes this process stable and viable.
vs GStyle: GStyle introduces limited geometric changes through Gaussian splitting, but this is essentially a preprocessing step rather than part of the optimization objective. This paper treats geometric optimization as a core optimization phase and explicitly guides it via GCFM.
vs StylizedGS / ABCGS: Both constrain geometry using depth-preservation losses, which is essentially about "preventing geometric changes" rather than "using geometry to express style." This paper does the opposite, actively utilizing depth and edge features to align geometry with the target style.
vs CLIPGaussian: CLIPGaussian supports joint optimization of color and geometry but prioritizes appearance adaptation, leaving geometric stylization relatively insufficient. This paper delves deeper and achieves better stability in geometric stylization using decoupled optimization and GCFM.

Rating¶

Novelty: ⭐⭐⭐⭐ Treats geometry as a "first-class citizen" in 3DGS style transfer for the first time instead of a protected object; the combination of decoupled optimization + GCFM is simple and effective. However, decoupled alternating optimization and contrastive feature matching are not entirely new concepts individually; the innovation lies primarily in their combination and adaptation to the geometric sensitivity of 3DGS.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 72 scene-style combinations, 5 baselines, SIFID + multi-view consistency + user study + inference time + GCFM ablation + optimization strategy ablation + update ratio ablation + hyperparameter sensitivity, covering a comprehensive and in-depth range.
Writing Quality: ⭐⭐⭐⭐ Clearly structured. The visual comparison of joint vs. decoupled optimization in Fig. 3 very intuitively illustrates the core motivation. The supplementary materials are rich (algorithmic pseudo-code + scene-by-scene efficiency + user study details + extra qualitative comparisons). Some formulas contain numerous symbols, but readability remains acceptable.
Value: ⭐⭐⭐⭐ Opens up the direction of 3DGS geometric stylization, which was previously avoided by the community, providing a solid baseline and technical path for subsequent works (such as controllable geometric stylization and text-to-3D geometric guidance). The code is open-source and highly practical.