Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting

Conference: ECCV 2026
arXiv: 2606.26754
Project Page: https://vcc2310.github.io/SceneStyler/
Code: None
Area: 3D Vision
Keywords: 3D Gaussian Splatting, style transfer, optimal transport, multi-view consistency, neural style transfer

TL;DR

This paper proposes a multi-view stylization framework for 3D Gaussian Splatting based on capacity-controlled optimal transport. It replaces greedy nearest-neighbor matching with semi-balanced optimal transport, suppressing the many-to-one problem through column capacity constraints. Simultaneously, it introduces cross-view match guidance to ensure style consistency between adjacent views, assisted by geometric regularization to enhance reconstruction quality. The proposed method comprehensively outperforms existing approaches in style quality, content fidelity, and multi-view consistency.

Background & Motivation

With its explicit and efficient scene representation capability, 3D Gaussian Splatting (3DGS) has become the core paradigm for novel view synthesis, naturally extending to 3D stylization tasks—transferring the style of a 2D artistic reference image to a 3D scene while preserving its content structure. However, ensuring multi-view style consistency remains a fundamental challenge for this task.

Existing 3DGS stylization methods (such as StylizedGS, SGSST, ABC-GS, StyleGaussian, etc.) generally adopt the paradigm of "independently applying 2D style losses view-by-view." This paradigm has severe limitations on two levels: (1) intra-view, the standard nearest-neighbor feature matching (NNFM) suffers from the many-to-one problem, where a large number of content features are matched to a few dominant style features, leading to repetitive textures and poor style diversity; (2) inter-view, when the same 3D scene content is projected to different views, differences in view-dependent features might cause them to match different style patterns, resulting in blurry or inconsistent styles when observed from novel viewpoints.

Recent works (such as MultiStyleGS and StylizedGS) attempt to improve consistency using DINO feature enhancement or depth-preserving losses. However, these methods only impose constraints "alongside" the matching process, without addressing the matching mechanism itself—failing to either prevent intra-view many-to-one matching or explicitly constrain the same 3D content to match consistent style patterns across views.

This paper directly addresses the root cause of the problem: reformulating style matching as a capacity-constrained optimal transport problem, replacing greedy matching with a mathematically interpretable transport plan. Core Idea: Column capacity constraints of semi-balanced optimal transport are used to actively suppress many-to-one matching (controlling style diversity via a tunable parameter τ), while cross-view guidance maps propagate the matching preferences of adjacent views to the current view, achieving explicit constraint of matching consistency.

Method

Overall Architecture

The proposed method consists of two stages: The first stage is enhanced reconstruction, which introduces depth, shape, and scale geometric regularization terms in 3DGS training to make the Gaussian primitives smaller and more uniform, establishing a solid geometric foundation for stylization. The second stage is capacity-controlled stylization. After view-by-view rendering, the CCFT loss based on semi-balanced optimal transport replaces the NNFM loss, and a cross-view guidance map is integrated into the K-nearest neighbor search to stabilize the sparse matching support set. Finally, content loss is applied to constrain structural fidelity, and only color parameters are optimized to obtain the finalized stylized scene.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Images<br/>+ Depth Estimation"] --> B["Enhanced Reconstruction<br/>Depth/Shape/Scale Regularization"]
    B --> C["Enhanced 3DGS"]
    C --> D["CCFT Loss<br/>Semi-Balanced Optimal Transport"]
    E["Style Reference Image"] --> D
    D --> F["Cross-View Match Guidance<br/>Guidance Map Aggregation & Distance Modification"]
    F -->|"Previous View<br/>Transport Matrix"| D
    D --> G["Content Loss<br/>Optimize Color Parameters Only"]
    G --> H["Stylized 3D Scene"]

Key Designs

1. Enhanced Reconstruction: Establishing a Geometric Foundation for Stylization

To efficiently fit high-frequency textures in real views, vanilla 3DGS often optimizes highly irregular and highly anisotropic Gaussian primitives. These elongated ellipsoids produce geometric artifacts during stylization, while large-scale primitives occupy large volumes with low density, failing to carry delicate style textures. Prior works (e.g., StylizedGS, ABC-GS) only passively filter out low-quality Gaussians after reconstruction, without correcting the geometry at its source.

This paper introduces three regularization terms during the reconstruction stage: depth loss \(\mathcal{L}_{\mathit{depth}} = \|D_{\mathit{render}} - D_{\mathit{pred}}\|_1\), which uses a pretrained depth estimator (DAv2) to provide supervision, suppress floaters, and concentrate Gaussians on valid object surfaces; shape regularization \(\mathcal{L}_{\mathit{shape}} = \frac{1}{N_{gs}}\sum [S_{\mathit{max}}^i - \lambda_s S_{\mathit{min}}^i]_+\), which constrains the ratio of the longest axis to the shortest axis to not exceed \(\lambda_s=3.5\) to suppress needle-like distorted Gaussians; and scale regularization \(\mathcal{L}_{\mathit{size}} = \frac{1}{N_{gs}}\sum ([S_{\mathit{max}}^i - \beta_1]_+ + [\beta_2 - S_{\mathit{max}}^i]_+)\), which limits the longest axis within the range of \([\beta_2, \beta_1]=[0.001, 0.1]\). The total reconstruction loss is \(\mathcal{L}_{\mathit{rec}} = \mathcal{L}_{\mathit{photo}} + \lambda_{\mathit{depth}}\mathcal{L}_{\mathit{depth}} + \lambda_{\mathit{shape}}\mathcal{L}_{\mathit{shape}} + \lambda_{\mathit{size}}\mathcal{L}_{\mathit{size}}\), where \(\lambda_{\mathit{depth}}\) exponentially decays from 1 to 0.01.

Effect: The depth loss stabilizes the global spatial structure, the shape regularization ensures the Gaussians uniformly cover the surface, and the scale regularization refines local details. The combination of these three components provides a "clean canvas" for the stylization stage, allowing fine-grained style textures to be faithfully attached to geometrically accurate surfaces.

2. CCFT Loss: Replacing Greedy Nearest-Neighbor Matching with Semi-Balanced Optimal Transport

The core limitation of the NNFM loss is that the matching is "winner-take-all": each content feature greedily selects the style feature with the highest cosine similarity, causing a few dominant style features to be repeatedly used (many-to-one), resulting in large-area repetitive textures.

This paper reformulates it as a semi-balanced optimal transport problem. Let the rendered features be \(X=\{x_i\}_{i=1}^N\) and the style features be \(Y=\{y_j\}_{j=1}^M\). For each \(x_i\), the \(K=16\) style features with the highest cosine similarity are found to construct a sparse support set \(E=\{(i,j) \mid j \in \mathrm{KNN}(i)\}\). On this bipartite graph, the semi-balanced optimal transport with entropy regularization is solved:

\[T = \arg\min_{T \ge 0} \left[\langle T,C\rangle_E - \varepsilon H(T)_E + \tau\,\mathrm{KL}(T^\top \mathbf{1} \| b)\right], \quad \text{s.t. } T\mathbf{1}=a,\ a_i=\frac{1}{N}\]

where \(C\) is the cosine distance cost matrix, \(H(T)\) is the entropy regularization (controlling smoothness, \(\varepsilon=0.05\)), and \(\mathrm{KL}(T^\top\mathbf{1}\|b)\) is the soft column capacity constraint—\(b\) is a uniform distribution over the reachable style features (\(b_j = 1/|J|\) if \(j\) is selected as a neighbor by at least one content feature, and \(10^{-12}\) otherwise), and \(\tau=0.5\) controls the strength of the capacity constraint.

This is iteratively solved via the Sinkhorn-Knopp algorithm (where the main difference in the update rule from standard Sinkhorn is that the column update becomes \(v^{(t+1)} = (b / (K^\top u^{(t+1)}))^{\frac{\tau}{\tau+\varepsilon}}\), i.e., a soft projection). The final CCFT loss is the total cost under the optimal transport plan, \(\mathcal{L}_{\mathit{CCFT}} = \langle T,C\rangle_E\). Core Mechanism: when \(\tau\to 0\), it degenerates into NNFM (greedy matching); when \(\tau\to\infty\), a strictly uniform allocation is enforced. Thus, \(\tau\) acts as the "knob" for style diversity. Paper experiments show that \(\tau=0.01\) produces large patches of repetitive stripe textures, \(\tau=0.1\) results in swirl textures, and \(\tau=1\) reveals richer circular details, validating the controllability of capacity constraints over style diversity.

3. Cross-View Match Guidance: Propagating Matching Preferences Across Views with a Guidance Map

While CCFT addresses the intra-view many-to-one problem, the challenge of cross-view consistency lies in: when searching for K-nearest neighbors under a new view at each iteration, the feature space shifts due to view changes, causing features of the same 3D content to be associated with different style features. This instability of the KNN graph leads to matching drift.

This paper introduces a cross-view guidance map mechanism. Core Operation: in the KNN search for view \(v\), the pure content-style distance is replaced by the guided distance:

\[\mathrm{Distance}(x_i^{(v)}, y_j) = \mathrm{Sim}(x_i^{(v)}, y_j) + \lambda_{\mathit{guide}}\,\mathrm{Sim}(x_i^{(v)}, g_j^{(v)})\]

where the guidance map \(g_j^{(v)} = \sum_i T_{ij}^{(v-1)} x_i^{(v-1)}\) is an L2-normalized vector obtained by aggregating the content features of the previous view, weighted by the transport matrix—it encodes "which content features in the previous frame were matched to style feature \(j\)." The function of the guidance map is: if a content feature in the current view is similar to \(g_j^{(v)}\) in the guidance map space, it indicates that it also "resembles" those content features matched to \(j\) in the previous frame, and thus should also be matched to \(j\). This achieves a soft propagation of matching preferences across views.

Note: The guidance map only affects the construction of the KNN graph (determining \(E\)) and does not affect the transport cost \(C\)—\(C\) is always computed using the pure content-style cosine distance, ensuring that style matching quality is not compromised by the guidance intensity. The best balance between consistency and content fidelity is achieved when \(\lambda_{\mathit{guide}}=1\).

A Complete Example

Assume a scene has 3 adjacent training views \(v_1, v_2, v_3\). At \(v_1\), rendered features \(X^{(1)}\) (\(N=1024\) spatial locations) and style features \(Y\) (\(M=512\)) are extracted from the VGG relu3_1 layer. For each \(x_i^{(1)}\), a \(K=16\)-nearest neighbor search is performed to obtain a sparse edge set \(E\), and 5 rounds of Sinkhorn iterations are executed to yield the transport matrix \(T^{(1)}\). At this point, \(\mathcal{L}_{\mathit{CCFT}} = \sum T_{ij} C_{ij}\). Subsequently, the guidance map \(G^{(2)}\) is computed: for each style feature \(j\), \(g_j^{(2)}\in\mathbb{R}^C\) is the weighted sum of all content features in \(v_1\) that were matched to \(j\) (after L2-normalization). When entering \(v_2\), the KNN search no longer solely considers the similarity between \(x_i^{(2)}\) and \(y_j\), but employs the guided distance—if \(x_i^{(2)}\) happens to correspond to the region in \(v_1\) that matched the "ocean wave texture" style feature, it will also be close to \(g_{\text{海浪}}^{(2)}\) in the guidance map space, thereby being automatically directed to the same style feature. The same applies to \(v_3\). After 50 iterations of optimization per view (for forward-facing scenes), all views eventually converge to consistent and diverse style allocations.

Loss & Training

The total loss during the stylization stage is \(\mathcal{L}_{\mathit{style}} = \lambda_{\mathit{CCFT}}\mathcal{L}_{\mathit{CCFT}} + \lambda_{\mathit{content}}\mathcal{L}_{\mathit{content}}\), where \(\mathcal{L}_{\mathit{content}} = \frac{1}{N}\sum (x_i - \hat{x}_i)^2\) computes the MSE distance between the rendered features and the original scene features, \(\lambda_{\mathit{CCFT}}=30\), and \(\lambda_{\mathit{content}}=0.005\). Key design choice of training strategy: during the stylization stage, all geometric parameters (position, covariance, opacity) are frozen, and only spherical harmonics coefficients (color parameters) are optimized—the ablation comparison with optimizing all parameters shows that full-parameter optimization causes structural loss to surge from 0.0318 to 0.0607 (where the scene geometry is heavily distorted by the style loss), while optimizing color parameters alone perfectly preserves the original 3D geometric structure while transferring the style. Furthermore, color transfer (post-color transfer, following the practice of ARF) is applied once both before and after stylization to maintain color tone consistency with the reference image. For forward-facing scenes, 50 iterations per view are performed (total iterations >= 2000), while 360-degree scenes require 15-20 iterations per view. Using a single RTX 4090 GPU, the stylization takes an average of 8.35 minutes.

Key Experimental Results

Main Results

On the LLFF (forward-facing), T&T, and Mip-NeRF 360 (360-degree) datasets, 5 scenes x 6 style reference images = 60 test cases are evaluated. Baseline methods: StylizedGS, ABC-GS, SGSST, CLIPGaussian, StyleGaussian.

Method	ArtFID↓	Structure Loss↓	Short MEt3R↓	Long MEt3R↓
StylizedGS	23.068	0.0462	0.1201	0.2858
ABC-GS	25.026	0.0357	0.1198	0.2822
SGSST	25.143	0.0568	0.1384	0.3064
CLIPGaussian	27.293	0.0431	0.1353	0.2903
StyleGaussian	31.855	0.0425	0.1284	0.2982
Ours (3DGS)	24.133	0.0320	0.1198	0.2795
Ours	22.801	0.0318	0.1196	0.2795

User study (25 participants, each evaluating 6 scenes, totaling 450 rankings): In terms of style quality, the proposed method achieves a 32.22% preference rate (followed by SGSST at 25.04%), 37.61% for content fidelity (followed by ABC-GS at 19.04%), and 37.61% for multi-view consistency (followed by ABC-GS at 19.83%), significantly leading in all three categories.

Ablation Study

Config	ArtFID↓	Structure Loss↓	Short MEt3R↓	Long MEt3R↓	Note
Full model (\(\lambda_{\mathit{guide}}=1\))	22.801	0.0318	0.1196	0.2795	full model
w/o Cross-View Guidance (\(\lambda_{\mathit{guide}}=0\))	22.809	0.0312	0.1204	0.2807	Both short-range and long-range consistency degrade after removing guidance
w/o Enhanced Reconstruction (using vanilla 3DGS)	24.133	0.0320	0.1198	0.2795	ArtFID significantly deteriorates (24.133 vs 22.801)
Optimize all parameters (not only color)	23.215	0.0607	0.1358	0.2905	Structural loss explodes, multi-view consistency collapses
Unordered view training	22.806	0.0315	0.1202	0.2804	Guidance effect is significantly weakened, approaching the unguided version
Increase guidance intensity (\(\lambda_{\mathit{guide}}=10\))	22.805	0.0323	0.1191	0.2786	Consistency continues to improve, but content structure is slightly compromised

Progressive ablation of geometric regularization (visualization): With only \(\mathcal{L}_{\mathit{photo}}\), the scene geometry is cluttered; adding \(\mathcal{L}_{\mathit{depth}}\) stabilizes the global structure; adding \(\mathcal{L}_{\mathit{shape}}\) suppresses needle-like distorted Gaussians; and adding \(\mathcal{L}_{\mathit{size}}\) refines local details—the combination of these four components transforms the Gaussian primitives from cluttered and disordered to uniform and fine, providing a more stable geometric foundation for stylization.

Loss function ablation: Replacing CCFT with NNFM Loss (only matching monotonous wave textures in the background, resulting in large-area repetition), FAST Loss (capturing some style patterns but lacking details and generating artifacts), or GRAM Loss (failing to transfer local style patterns, with severe artifacts) shows that CCFT is the only choice that simultaneously achieves style details and distribution uniformity.

Key Findings

• CCFT loss is the core contribution: The column capacity constraint mathematically guarantees the suppression of many-to-one matching, and the \(\tau\) parameter provides a continuous and controllable spectrum from "greedy matching" to "uniform distribution," which is unachievable by NNFM, FAST, or GRAM.
• A 2D-3D tension exists in cross-view guidance: Increasing \(\lambda_{\mathit{guide}}\) improves consistency (Long MEt3R decreases from 0.2807 to 0.2786), but slightly compromises content structure (Structure Loss increases from 0.0312 to 0.0323)—essentially, 2D style patterns overwrite some 3D content boundaries. This parameter requires a trade-off in practical applications.
• Enhanced reconstruction is an important auxiliary contribution: Using only vanilla 3DGS, the ArtFID of Ours (3DGS) is 24.133. Incorporating enhanced reconstruction improves it to 22.801, a significant margin—demonstrating that geometric quality directly limits the upper bound of stylization.
• Optimizing only colors is the most underestimated design choice: Full-parameter optimization leads to a structural loss of 0.0607, whereas color optimization yields only 0.0318—a nearly twofold difference, proving that freezing geometry during the stylization stage is necessary.

Highlights & Insights

• Replacing nearest-neighbor matching with optimal transport: Elevating style matching from an "undifferentiable, non-global" greedy search to a "differentiable, globally constrained" transport optimization problem represents a conceptual dimensionality upgrade. The column capacity constraint directly models the many-to-one problem without workarounds, offering a clean theoretical formulation.
• Guidance map propagation is a reusable design pattern: The core of cross-view guidance lies in "encoding the optimal transport plan of the previous frame into a guidance map and injecting it into the KNN retrieval of the next frame"—this pattern of "softly guiding the current search space with historical optimal solutions" is not only suitable for view consistency but can also be transferred to other scenarios such as inter-frame consistency in video and multimodal feature alignment.
• Sparsification strategy of Sinkhorn reduces memory: Keeping only Top-K nearest-neighbor edges on the bipartite graph (instead of fully connected) reduces the OT computation from \(O(NM)\) to \(O(NK)\). While maintaining transport quality, it controls peak GPU memory to 8.37GB (lower than ABC-GS's 9.63GB), representing a practical engineering optimization.
• Two-stage decoupling of geometric regularization and stylization: Reconstructing and optimizing the geometry first, and then freezing the geometry while only optimizing color for stylization—this decoupling prevents style loss from "contaminating" the geometry. Possessing a simple yet highly effective logic, this approach can be generalized to other 3D editing tasks.

Limitations & Future Work

• The 2D-3D tension in cross-view guidance is an inherent contradiction: Guidance essentially imposes consistency constraints in the 2D rendering domain, whereas the scene is 3D—when 2D style patterns are excessively aligned, they can erase 3D content boundaries. The authors acknowledge this trade-off and adopt \(\lambda_{\mathit{guide}}=1\) as a compromise in practice, but do not provide an adaptive adjustment mechanism. A potential direction for improvement is weighting the guidance intensity with depth information—decreasing \(\lambda_{\mathit{guide}}\) near geometric boundaries and increasing it in flat areas.
• Scene-by-scene optimization, unable to achieve real-time: Stylization takes 8.35 minutes per scene, which is far from real-time or interactive applications. The authors mention feed-forward methods as a future direction, but how to achieve a feed-forward model while maintaining capacity constraints and cross-view guidance remains an open problem.
• View order dependency: The guidance mechanism assumes that training views are ordered in a spatially continuous sequence (which is naturally satisfied by forward-facing and 360-degree datasets). A random view order significantly weakens the guidance effect (with Long MEt3R regressing to 0.2804). For custom data with sparse or non-uniform camera trajectories, camera interpolation or inserting virtual views is required to restore continuity.
• Only tested on VGG features: All experiments are based on the relu3_1 layer of VGG-16, without exploring more modern feature extractors such as DINO or CLIP. The impact of different backbone feature spaces on OT matching behavior and guidance map quality is a dimension that warrants further study.
• Style reference is restricted to a single image: Multi-style references or text-guided scene stylization have not been evaluated.

Related Work & Insights

• vs ARF (NNFM): ARF pioneered high-quality 3D stylization by introducing the NNFM loss, but its many-to-one nature leads to monotonous and repetitive textures. This paper replaces NNFM with CCFT, resolving this issue at its root through column capacity constraints—ARF can be seen as a special case of CCFT when \(\tau\to 0\).
• vs ABC-GS (FAST Loss): ABC-GS replaces NNFM with feature alignment to alleviate many-to-one, but global linear transformation produces mismatched results in regions with insufficient local semantics (e.g., matching sunflower seed patterns to a triceratops's mouth) and over-smoothes high-frequency features. The proposed OT method preserves local semantic accuracy while providing explicit capacity control.
• vs StylizedGS / MultiStyleGS: The former enhances geometric consistency with depth-preserving loss, while the latter enhances feature uniqueness with DINO features—both being indirect methods. The cross-view guidance proposed in this work imposes constraints directly at the matching mechanism level, representing the first work to explicitly constrain the same 3D content to match consistent style patterns across views.
• vs SGSST (Gram Loss): The Gram matrix based on global statistics can cause structural collapse when local structure differences between content and style are large (e.g., making flower petals and leaves indistinguishable). The proposed local OT matching naturally preserves spatial correspondence.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introducing optimal transport into 3DGS stylization, the combination of semi-balanced OT column capacity constraint and cross-view guidance map shows genuine originality at the problem-modeling level, rather than being a simple stack of modules.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ The main experiments cover 5 baselines, 3 datasets, and 60 test cases. Ablations span 6 dimensions: loss function, capacity control, guidance intensity, geometric regularization, parameter optimization strategy, and view order. The user study includes 25 participants and 450 rankings. The only missing part is an analysis of different VGG layers and backbones.
• Writing Quality: ⭐⭐⭐⭐☆ Clear problem statement and complete motivation chain (many-to-one + inter-view inconsistency → two root causes → two corresponding mechanism solutions). The mathematical derivation is solid, and the appendix includes complete algorithm pseudocode. The only limitation is that the OT theoretical background is somewhat lengthy, and its connection to the core contributions could be tighter.
• Value: ⭐⭐⭐⭐⭐ Multi-view consistency in 3DGS stylization is a widely recognized core pain point in this field. This paper provides the first systematic solution at the matching mechanism level. The trio of CCFT loss, cross-view guidance, and enhanced reconstruction has clear reference value and transfer potential for future work.

Related Papers

• [ICLR 2026] Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting
• [ECCV 2026] Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures
• [ECCV 2026] ViewSplat: View-Adaptive 3D Gaussian Splatting for Feed-Forward Synthesis
• [ECCV 2026] Geometry-Aware Style Transfer in 3D Gaussian Splatting
• [ECCV 2026] Monte Carlo Energy Aggregation for Mobile 3D Gaussian Splatting