Geometry-Aware Style Transfer in 3D Gaussian Splatting¶

Conference: ECCV 2026
arXiv: 2606.24144
Code: https://github.com/oweixx/gast
Area: 3D Vision
Keywords: 3D Gaussian Splatting, style transfer, decoupled optimization, contrastive learning, geometry-aware

TL;DR¶

This paper proposes a geometry-aware 3DGS style transfer framework that alternately updates color and geometry parameters via decoupled optimization. It introduces a multimodal (RGB + depth + edge) contrastive feature matching loss (GCFM) to guide the transfer of style structures and geometric textures, significantly outperforming existing 3DGS style transfer methods in style fidelity and multi-view consistency.

Background & Motivation¶

2D image style transfer has become highly mature, with successful solutions ranging from Gatys' Gram matrix to diffusion models, but they cannot guarantee cross-view consistency. With the rise of NeRF and 3DGS, 3D style transfer has become feasible—among which 3DGS, with its explicit geometric primitives and real-time rendering capabilities, has become the dominant backbone in this direction.

However, existing 3DGS style transfer methods generally adopt a conservative strategy: they only transfer color/appearance attributes while applying strong constraints to keep geometric parameters unchanged. The reason is that 3DGS rendering results are highly sensitive to changes in geometric parameters—color modifications only affect the surface appearance, whereas directly modifying geometric parameters (position, covariance, opacity) alters the underlying structure of the scene, and even minor geometric miscalculations can severely degrade visual quality. This is like applying wood-grain wallpaper to furniture: the wallpaper can conform to the shape but can never change the structure to reflect the actual bump textures and structural irregularities of real wood.

This creates a Key Challenge: geometry is an important medium of style (such as the thickness of Van Gogh's brushstrokes or the block-plane structures of Cubism), but directly optimizing 3DGS geometric parameters easily leads to scene collapse. Existing methods either only optimize color (StyleGaussian, SGSST), perform limited geometric adjustments only in the preprocessing stage (Gaussian splitting in GStyle), or constrain geometry to remain unchanged using depth preservation losses (StylizedGS, ABCGS)—essentially avoiding geometric stylization rather than treating geometry as a medium for style expression.

Core Idea: Treat color and geometry as two equal dimensions of stylization, eliminate their update interference through decoupled alternating optimization, and guide the geometric structure to align with the target style using multimodal contrastive feature matching (simultaneously considering RGB, depth, and edges).

Method¶

Overall Architecture¶

The goal of this paper is to transfer the artistic style of a 2D style image \(\mathbf{I}^{\mathcal{S}}\) to a 3DGS scene reconstructed from a set of multi-view content images \(\mathbb{I}^{\mathcal{C}}\), while maintaining spatial and geometric consistency. The overall pipeline consists of three steps:

First, use the standard 3DGS pipeline to reconstruct the initial Gaussian representation \(\mathbb{G}^{\text{init}}\) from the content images. Second, apply a linear transformation to the Gaussian colors using the color matching from ARF to align the initial color distribution with the target style, yielding \(\mathbb{G}^{\text{c}}\). The third step is the core—alternately performing the color optimization phase and the geometry optimization phase in the outer loop via decoupled optimization, where both phases are driven by the proposed GCFM loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Content Images"] --> B["Initial 3DGS Reconstruction"]
    B --> C["Color Matching<br/>Linear Transformation to Align Style Tones"]
    C --> D["Color Optimization Phase<br/>Fix Geometry, Update DC Colors"]
    D --> E["Geometry Optimization Phase<br/>Fix Colors, Update Position/Covariance/Opacity"]
    E -->|K Outer Loops| D
    E --> F["Stylized 3DGS Scene"]
    G["Style Image"] --> H["Multimodal Feature Extraction<br/>RGB + Depth + Edge"]
    H --> I["GCFM Contrastive Loss<br/>Pull Positive / Push Negative Samples"]
    I -.->|Drive Both Phases| D
    I -.->|Drive Both Phases| E

Key Designs¶

1. Decoupled Optimization: Alternately Updating Color and Geometry to Eliminate Joint Optimization Interference

Existing methods either only optimize color while freezing geometry, or jointly optimize both but face severe optimization conflicts. During joint optimization (joint-step), gradients for color and geometry propagate simultaneously, easily leading to color over-stylization—the optimization process erroneously modifies geometry to embed low-level style texture patterns, which in turn destroys the scene structure (e.g., the flower structure in Fig. 3 gradually collapsing with iterations).

The decoupled optimization proposed in this paper divides the complete optimization process into \(K\) outer loops, sequentially solving two sub-problems in each loop:

\[\boldsymbol{\Theta}^{k+1} = \arg\min_{\boldsymbol{\Theta}} \mathcal{L}(\boldsymbol{\Theta}, \boldsymbol{\Phi}^k) \quad \text{(颜色阶段)}\]

\[\boldsymbol{\Phi}^{k+1} = \arg\min_{\boldsymbol{\Phi}} \mathcal{L}(\boldsymbol{\Theta}^{k+1}, \boldsymbol{\Phi}) \quad \text{(几何阶段)}\]

where \(\boldsymbol{\Theta}\) represents the color parameters (only the zero-order DC component of spherical harmonics, i.e., basic RGB, is optimized, while high-order coefficients remain fixed), and \(\boldsymbol{\Phi} = \{\boldsymbol{\mu}_n, \boldsymbol{\Sigma}_n, \alpha_n\}\) represents the geometric parameters (position, covariance, opacity). In practice, each sub-problem is approximately solved using a fixed number of gradient descent steps: \(N_c = 10\) steps for the color stage, \(N_g = 90\) steps for the geometry stage, and \(K = 30\) outer loops, totaling 3000 iterations.

Why it works: The essence of alternating optimization is establishing a "mutual guidance loop"—updating the color first aligns the appearance with the target style, providing a stable visual reference for subsequent geometric updates; then, when updating the geometry, having a reliable color foundation allows the model to more accurately determine which structural changes are required by the style rather than being noise. Ablation studies also confirm that the SIFID of joint optimization (1.4220) is significantly worse than that of decoupled optimization (1.1736) and suffers from scene collapse induced by color over-stylization.

2. GCFM: Multimodal Contrastive Feature Matching via Jointly Guiding Geometric Style Alignment with RGB, Depth, and Edges

Traditional feature matching in 3D style transfer (such as ARF) usually performs nearest neighbor matching only in the RGB domain, lacking awareness of geometric structures. The proposed GCFM captures style features from three dimensions:

Color features \(\mathbf{F}_{\text{color}}\): Standard RGB features extracted from VGG, capturing texture and color styles.
Depth features \(\mathbf{F}_{\text{depth}}\): The depth of the rendered image comes directly from the 3DGS rendering pipeline; the depth of the style image is estimated by DepthAnythingV2. Depth provides coarse-grained spatial structural information.
Edge features \(\mathbf{F}_{\text{edge}}\): Extracted using the Canny operator on depth maps rather than color images (as style images have complex textures and high noise in color edges). Edges capture structural boundaries and guide contour-level shape alignment.

The three-modal features are concatenated channel-wise into a joint feature map \(\mathbf{F}^s = [\mathbf{F}_{\text{color}}^s, \mathbf{F}_{\text{depth}}^s, \mathbf{F}_{\text{edge}}^s]\) (where \(s \in \{\mathcal{R}, \mathcal{S}\}\) denotes the rendered image and the style image).

Unlike prior works that only use positive sample (nearest neighbor) matching, GCFM introduces a contrastive learning mechanism: for each spatial location \(\mathbf{x}\) in the rendered feature map, it finds the most similar positive sample \(\mathbf{y}^+(\mathbf{x})\) and the least similar negative sample \(\mathbf{y}^-(\mathbf{x})\) in the style feature map:

\[\mathbf{y}^+(\mathbf{x}) = \arg\min_{\mathbf{y}} D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}))\]

\[\mathbf{y}^-(\mathbf{x}) = \arg\max_{\mathbf{y}} D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}))\]

where \(D_{\cos}(\cdot, \cdot) = 1 - \cos(\cdot, \cdot)\) represents the cosine distance. The GCFM loss pulls the rendered features (anchors) toward positive samples and pushes them away from negative samples:

\[\mathcal{L}_{\text{GC}} = \frac{1}{|\Omega_{\mathbf{F}}|} \sum_{\mathbf{x} \in \Omega_{\mathbf{F}}} \Big(2 + D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}^+(\mathbf{x}))) - D_{\cos}(\mathbf{F}^{\mathcal{R}}(\mathbf{x}), \mathbf{F}^{\mathcal{S}}(\mathbf{y}^-(\mathbf{x})))\Big)\]

Ablation studies show that removing the contrastive objective (using only positive sample matching) leads to blurry structural features in the depth map; using a single modality (RGB only) over-simplifies geometric contours. The full GCFM (trimodal + contrastive) yields the sharpest geometric contours and the most faithful reproduction of stylized structures.

Loss & Training¶

The total loss function is a weighted sum of four terms, with different terms used in the color phase and geometry phase:

\[\mathcal{L} = \lambda_{\text{GC}} \mathcal{L}_{\text{GC}} + \lambda_{\text{cont}} \mathcal{L}_{\text{cont}} + \lambda_{\text{TV}} \mathcal{L}_{\text{TV}} + \delta_{\text{reg}} \mathcal{L}_{\text{reg}}\]

GCFM loss \(\mathcal{L}_{\text{GC}}\) (\(\lambda_{\text{GC}} = 2.0\)): The aforementioned multimodal contrastive feature matching loss, which serves as the core driving term for style transfer.
Content preservation loss \(\mathcal{L}_{\text{cont}}\) (\(\lambda_{\text{cont}} = 1 \times 10^{-3}\)): VGG-16 perceptual loss, which constrains the feature distance between the rendered image and the content image to ensure the scene structure is not compromised.
TV regularization \(\mathcal{L}_{\text{TV}}\) (\(\lambda_{\text{TV}} = 0.02\)): Restricts local spatial variations in horizontal and vertical directions of the rendered image, suppressing high-frequency noise and boundary artifacts.
Geometric regularization \(\mathcal{L}_{\text{reg}}\) (only enabled in the geometry stage, \(\delta_{\text{reg}} = 1\)): Constrains the optimized opacity \(\boldsymbol{\alpha}\), scale \(\mathbf{s}\), and rotation \(\mathbf{r}\) from deviating too far from their color-matched initial states, and additionally constrains the rendered depth map \(\mathbf{D}^{\mathcal{R}}\) to match the reference depth map \(\mathbf{D}^{\text{c}}\), preventing Gaussian inflation, collapse, or global blurring.

The optimizer is Adam with a learning rate of \(\eta = 1 \times 10^{-4}\) (reduced to \(2.5 \times 10^{-5}\) on the T&T dataset). Training is conducted on a single RTX A6000 (48GB), taking an average of approximately 14 minutes per scene stylization.

Key Experimental Results¶

Main Results¶

8 scenes (4 from LLFF, 2 from T&T, and 2 from MipNeRF-360) and 9 style images, yielding a total of 72 scene-style combinations. SIFID is used to evaluate style fidelity (lower is better), and short-range/long-range consistency are measured using optical flow warping + masked RMSE/LPIPS.

Method	SIFID ↓	Short-range LPIPS ↓	Short-range RMSE ↓	Long-range LPIPS ↓	Long-range RMSE ↓
StyleGaussian	3.3405	0.050	0.053	0.141	0.114
G-Style	1.2955	0.045	0.035	0.118	0.085
StylizedGS	2.2439	0.028	0.021	0.072	0.062
SGSST	1.3936	0.048	0.048	0.129	0.109
CLIPGaussian	3.3685	0.048	0.046	0.128	0.108
Ours	1.1736	0.042	0.034	0.114	0.084

The proposed method achieves the best performance in SIFID (1.1736), significantly leading the second-best G-Style (1.2955). In terms of multi-view consistency, after excluding StylizedGS which has artificially inflated metrics due to over-smoothing/desaturation, the proposed method ranks first in both short-range and long-range LPIPS/RMSE. Meanwhile, in a 37-participant user study, it scored the highest in style similarity, visual appeal, and content recognizability.

The average stylization time is 14 minutes and 14 seconds, which is significantly faster than StyleGaussian (410 minutes) and SGSST (59 minutes), achieving the best trade-off between speed and quality.

Ablation Study¶

Ablation of Optimization Strategies (Table 3): This compares the impact of four strategies—geometry-only optimization, color-only optimization, joint optimization (joint-step), and decoupled optimization—on SIFID and consistency.

Config	SIFID ↓	Short-range LPIPS ↓	Short-range RMSE ↓	Long-range LPIPS ↓	Long-range RMSE ↓
Geometry-only	2.5496	0.045	0.040	0.113	0.096
Color-only	1.5392	0.042	0.037	0.115	0.086
Joint optimization	1.4220	0.042	0.036	0.122	0.087
Decoupled optimization (Ours)	1.1736	0.042	0.034	0.114	0.084

Geometry-only optimization yields the worst SIFID (2.5496) because the lack of color adaptation prevents effective matching of style distribution; color-only optimization improves SIFID but is still limited (1.5392) as it cannot capture geometric style cues; joint optimization achieves a moderate SIFID (1.4220) but performs worst in long-range consistency due to color over-stylization; decoupled optimization achieves the best performance across all metrics, proving that separating color and geometry updates stabilizes optimization and achieves a harmonious fusion of appearance and geometry.

GCFM Ablation: Removing the contrastive objective led to blurry structural features in the depth map; using a single modality (RGB only) resulted in over-simplified geometric contours; the complete GCFM produced the sharpest geometric contours and the most faithful structural reproduction. The SIFID was stable (1.1683-1.1878) across a GCFM weight \(\lambda_{\text{GC}}\) range of 1.0-10.0, demonstrating good robustness.

Ablation of Geometry/Color Update Ratio: \(N_g = 90, N_c = 10\) achieves the best SIFID (1.1736); \(N_g = 100\) (pure geometry) is overly rigid and lacks color adaptation, while \(N_g = 30\) exhibits insufficient geometric stylization.

Key Findings¶

Decoupled optimization contributes the most: compared to joint optimization, it reduces SIFID from 1.4220 to 1.1736, while resolving the issues of color over-stylization and scene collapse in joint optimization.
Geometry optimization is not optional: color-only optimization yields a SIFID of 1.5392, which improves to 1.1736 when geometry optimization is incorporated, indicating that geometry is indeed an important dimension of style expression.
The multimodal design of GCFM is crucial: complementary depth + edge information enables structure-aware feature matching, while RGB alone cannot capture geometric style.
Geometric updates need to dominate (\(N_g : N_c = 90 : 10\)), reflecting that geometric stylization requires more delicate iterations than color stylization.

Highlights & Insights¶

Decoupling implies stability: Gradient conflicts naturally exist between color and geometry optimization, and alternating updates is the simplest yet highly effective solution—this "divide and conquer" approach is instructive for any 3D task involving the joint optimization of multiple parameter groups, without necessarily requiring complex gradient modulation or adversarial training.
Multimodal feature concatenation for contrastive matching: Channel concatenation of RGB + depth + edges is simple and direct, but extracting depth from the rendering pipeline and edges from the depth map is a well-designed detail—this avoids the pitfall of large noise in color edges caused by complex style image textures, reflecting careful consideration of "which information should be extracted from which source".
Explicitly introducing negative samples in contrastive learning: Previous feature matching methods only utilized nearest neighbor positive samples, whereas GCFM explicitly pushes away the farthest negative samples, forcing the model to learn more discriminative style representations. This idea can be extended to other tasks requiring cross-domain feature alignment (such as 3D editing or reference-based control in text-to-3D).
Geometric regularization anchoring the initial state: Constraining parameters from deviating too far from the color-matched state during the geometric optimization phase is essentially a "trust initialization" strategy—allowing the model to explore geometric deformations under guidance rather than letting it deform unconstrainedly and lead to collapse.

Limitations & Future Work¶

Limitations not explicitly discussed by the authors: The method relies on DepthAnythingV2 to estimate the depth map of style images. For highly abstract/non-realistic styles (such as pure color blocks or minimalist lines), depth estimation may be inaccurate or even meaningless. In such cases, the depth and edge branches of GCFM might introduce noise rather than useful signals.
Although the stylization time is faster than most methods, it still requires about 14 minutes per scene, which is still far from real-time interaction. The Gaussian count per scene (0.54M-4.20M) poses a high demand on GPU memory (up to 39 GB).
The "degree" of geometric stylization is controlled by the target texture scale and the ratio \(N_g / N_c\), lacking more intuitive user-controllable parameters (such as a "geometric deformation strength" slider), which is less user-friendly for artists.
Future improvement ideas: (1) Replace fixed concatenation with learnable depth/edge weights, allowing the model to adaptively determine the importance of each modality under different styles; (2) Explore feed-forward geometric stylization networks to bypass the limitation of per-scene optimization; (3) Incorporate prior knowledge from diffusion models to guide geometric deformations at a more abstract semantic level, rather than being restricted to pixel-level feature matching.

vs ARF / SNeRF / CoARF (NeRF style transfer): These methods perform feature-matching style transfer on NeRF, but the implicit representation of NeRF leads to slow rendering and difficulties in geometric manipulation. This paper operates on explicit 3DGS primitives, naturally supporting real-time rendering and direct geometric modification.
vs StyleGaussian / SGSST: These methods only update color attributes and freeze geometry, simplifying style transfer to "reskinning 3D models." This paper demonstrates that geometry can also serve as a style carrier, and decoupled optimization makes this process stable and feasible.
vs GStyle: GStyle introduces limited geometric changes through Gaussian splitting, but this is essentially preprocessing rather than a part of the optimization objective. This paper treats geometric optimization as the core optimization stage, explicitly guided by GCFM.
vs StylizedGS / ABCGS: Both methods constrain geometry with depth preservation losses, which is essentially "preventing geometric changes" rather than "using geometry to express style." This paper does the opposite, actively guiding geometry to align with the target style using depth and edge features.
vs CLIPGaussian: CLIPGaussian supports joint optimization of color and geometry but emphasizes appearance adaptation, resulting in insufficient geometric stylization. This paper achieves deeper and more stable geometric stylization using decoupled optimization and GCFM.

Rating¶

Novelty: ⭐⭐⭐⭐ For the first time in 3DGS, geometry is treated as a "first-class citizen" of stylization rather than a protected object; the combination of decoupled optimization + GCFM is clean and effective. However, neither decoupled alternating optimization nor contrastive feature matching represents an entirely new concept; the innovation lies mainly in the combination strategy and adaptation to the sensitivity of 3DGS geometry.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 72 scene-style combinations, 5 baselines, SIFID + multi-view consistency + user study + inference time + GCFM ablation + optimization strategy ablation + update ratio ablation + hyperparameter sensitivity, covering comprehensive and in-depth dimensions.
Writing Quality: ⭐⭐⭐⭐ The structure is clear, and the visual comparison of joint vs. decoupled optimization in Fig. 3 very intuitively illustrates the core motivation. The supplementary material is rich (algorithm pseudocode + per-scene efficiency + user study details + additional qualitative comparisons). Some formulas contain many symbols but the readability is acceptable.
Value: ⭐⭐⭐⭐ This work opens the direction of 3DGS geometric stylization, which has been avoided by the community, providing a solid baseline and technical pipeline for subsequent work (such as controllable geometric stylization and text-to-3D geometric guidance). The code is open-source, offering high practicality.