Capacity-Controlled Multi-View Stylization of 3D Gaussian Splatting

Conference: ECCV 2026
arXiv: 2606.26754
Project Page: https://vcc2310.github.io/SceneStyler/
Code: None
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Style Transfer, Optimal Transport, Multi-View Consistency, Neural Style Transfer

TL;DR

This paper proposes a multi-view stylization framework for 3D Gaussian Splatting based on capacity-controlled optimal transport. By replacing greedy nearest-neighbor matching with semi-balanced optimal transport, it suppresses the "many-to-one" matching issue via column capacity constraints. Concurrently, it introduces a cross-view matching guidance mechanism to ensure style consistency between adjacent views, supported by geometric regularization to enhance reconstruction quality. The proposed method comprehensively outperforms existing approaches in style quality, content fidelity, and multi-view consistency.

Background & Motivation

3D Gaussian Splatting (3DGS), with its explicit and efficient scene representation capability, has become a core paradigm for novel view synthesis, and naturally extends to 3D stylization tasks—transferring the style of a 2D artistic reference image to a 3D scene while preserving its content structure. However, ensuring multi-view style consistency remains a fundamental challenge for this task.

Existing 3DGS stylization methods (StylizedGS, SGSST, ABC-GS, StyleGaussian, etc.) generally adopt the paradigm of "independently applying 2D style losses view-by-view." This paradigm suffers from severe limitations at two levels: (1) Intra-view: Standard nearest-neighbor feature matching (NNFM) exhibits a "many-to-one" problem, where a vast number of content features are matched to a few dominant style features, causing repetitive textures and poor style diversity; (2) Inter-view: When the same 3D scene content is projected to different views, it might be matched to different style patterns due to view-dependent feature variations, leading to style blurring or inconsistency when observed from novel angles.

Recent works (MultiStyleGS, StylizedGS) attempt to improve consistency using DINO feature enhancement or depth preservation losses. However, these are merely external constraints imposed "alongside" the matching process, and do not address the matching mechanism itself—they neither prevent intra-view many-to-one matching nor explicitly constrain the same 3D content to match consistent style patterns across views.

This paper directly addresses the root cause of these issues by remodeling style matching as a capacity-constrained optimal transport problem, replacing greedy matching with a mathematically interpretable transport plan. Core Idea: Use column capacity constraints in semi-balanced optimal transport to actively suppress many-to-one matching (controlling style diversity via an adjustable parameter \(\tau\)), while propagating matching preferences from adjacent views to the current view using a cross-view guidance graph to achieve explicit constraints on matching consistency.

Method

Overall Architecture

The proposed method operates in two stages: The first stage is enhanced reconstruction, which introduces three geometric regularizations (depth, shape, and scale) into 3DGS training to make Gaussian primitives smaller and more uniform, establishing a solid geometric foundation for stylization. The second stage is capacity-controlled stylization. After rendering view-by-view, the NNFM loss is replaced with the Capacity-Controlled Feature Transport (CCFT) loss based on semi-balanced optimal transport. Concurrently, a cross-view matching guidance map is incorporated into the K-Nearest Neighbor (KNN) search to stabilize the sparse matching support set. Finally, content loss is used to constrain structural fidelity, and only the color parameters are optimized to obtain the final stylized scene.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Images<br/>+ Depth Estimation"] --> B["Enhanced Reconstruction<br/>Depth/Shape/Scale Regularization"]
    B --> C["Enhanced 3DGS"]
    C --> D["CCFT Loss<br/>Semi-Balanced Optimal Transport"]
    E["Style Reference Image"] --> D
    D --> F["Cross-View Matching Guidance<br/>Guidance Map Aggregation & Distance Correction"]
    F -->|"Previous View<br/>Transport Matrix"| D
    D --> G["Content Loss<br/>Optimize Color Parameters Only"]
    G --> H["Stylized 3D Scene"]

Key Designs

1. Enhanced Reconstruction: Preparing the Foundation for Stylization with Geometric Regularization

To efficiently fit high-frequency textures in real views, vanilla 3DGS optimizes highly irregular and extremely anisotropic Gaussian primitives. Elongated ellipsoids produce geometric artifacts during stylization, and large-scale primitives cover huge spaces with low density, rendering them unable to carry delicate style textures. Prior works (StylizedGS, ABC-GS) only passively filter out unqualified Gaussians after reconstruction rather than correcting the geometry at the source.

This paper introduces three regularizations during the reconstruction stage: Depth loss \(\mathcal{L}_{\mathit{depth}} = \|D_{\mathit{render}} - D_{\mathit{pred}}\|_1\), supervised by a pre-trained depth estimator (DAv2) to suppress floaters and concentrate Gaussians on valid object surfaces; Shape regularization \(\mathcal{L}_{\mathit{shape}} = \frac{1}{N_{gs}}\sum [S_{\mathit{max}}^i - \lambda_s S_{\mathit{min}}^i]_+\), constraining the ratio of the longest axis to the shortest axis to not exceed \(\lambda_s = 3.5\) to suppress needle-like distorted Gaussians; Scale regularization \(\mathcal{L}_{\mathit{size}} = \frac{1}{N_{gs}}\sum ([S_{\mathit{max}}^i - \beta_1]_+ + [\beta_2 - S_{\mathit{max}}^i]_+)\), bounding the longest axis within the range \([\beta_2, \beta_1] = [0.001, 0.1]\). The total reconstruction loss is \(\mathcal{L}_{\mathit{rec}} = \mathcal{L}_{\mathit{photo}} + \lambda_{\mathit{depth}}\mathcal{L}_{\mathit{depth}} + \lambda_{\mathit{shape}}\mathcal{L}_{\mathit{shape}} + \lambda_{\mathit{size}}\mathcal{L}_{\mathit{size}}\), where \(\lambda_{\mathit{depth}}\) decays exponentially from 1 to 0.01.

Effect: Depth loss stabilizes the global spatial structure, shape regularization ensures uniform Gaussian coverage over surfaces, and scale regularization refines local details. Combining all three provides a "clean canvas" for the stylization stage, allowing fine-grained style textures to be faithfully attached to geometrically accurate surfaces.

2. CCFT Loss: Semi-Balanced Optimal Transport Replacing Greedy Nearest Neighbor Matching

The core flaw of the NNFM loss is that matching is "winner-takes-all": each content feature greedily selects the style feature with the highest cosine similarity, resulting in a few dominant style features being repeatedly used (many-to-one), which yields large-scale repetitive textures.

This paper reformulates this as a semi-balanced optimal transport problem. Given rendering features \(X=\{x_i\}_{i=1}^N\) and style features \(Y=\{y_j\}_{j=1}^M\), for each \(x_i\), the \(K=16\) style features with the highest cosine similarities are found to form a sparse support set \(E=\{(i,j) \mid j \in \mathrm{KNN}(i)\}\). An entropy-regularized semi-balanced optimal transport problem is solved over this bipartite graph:

\[T = \arg\min_{T \ge 0} \left[\langle T,C\rangle_E - \varepsilon H(T)_E + \tau\,\mathrm{KL}(T^\top \mathbf{1} \| b)\right], \quad \text{s.t. } T\mathbf{1}=a,\ a_i=\frac{1}{N}\]

where \(C\) is the cosine distance cost matrix, \(H(T)\) is the entropy regularization (controlling smoothness, \(\varepsilon=0.05\)), \(\mathrm{KL}(T^\top\mathbf{1}\|b)\) is the soft column capacity constraint, \(b\) is a uniform distribution over reachable style features (\(b_j = 1/|J|\) if \(j\) is chosen by at least one content feature as a neighbor, and \(10^{-12}\) otherwise), and \(\tau=0.5\) controls the strength of the capacity constraint.

Applying the Sinkhorn-Knopp algorithm iteratively (the difference from standard Sinkhorn is that the column update becomes \(v^{(t+1)} = (b / (K^\top u^{(t+1)}))^{\frac{\tau}{\tau+\varepsilon}}\), i.e., a soft projection), the final CCFT loss is computed as the total cost under the optimal transport plan: \(\mathcal{L}_{\mathit{CCFT}} = \langle T,C\rangle_E\). Core Mechanism: When \(\tau\to 0\), it degenerates to NNFM (greedy); when \(\tau\to\infty\), it forces a strictly uniform distribution. Thus, \(\tau\) acts as a "dial" for style diversity. Experiments demonstrate that \(\tau=0.01\) produces large blocks of repetitive stripe textures, \(\tau=0.1\) yields vortex textures, and \(\tau=1\) introduces richer circular details, validating the controllability of style diversity through capacity constraints.

3. Cross-View Matching Guidance: Propagating Matching Preferences Across Views with a Guidance Map

While CCFT solves the intra-view many-to-one issue, the challenge of cross-view consistency remains: during KNN search in a new view, the feature space shifts due to view changes. As a result, features from the same 3D content can be associated with different style features, causing instability in the KNN graph and matching drift.

This paper introduces a cross-view matching guidance mechanism. The core operations are as follows: in the KNN search for view \(v\), the pure content-style distance is replaced with a guided distance:

\[\mathrm{Distance}(x_i^{(v)}, y_j) = \mathrm{Sim}(x_i^{(v)}, y_j) + \lambda_{\mathit{guide}}\,\mathrm{Sim}(x_i^{(v)}, g_j^{(v)})\]

where the guidance map \(g_j^{(v)} = \sum_i T_{ij}^{(v-1)} x_i^{(v-1)}\) is the L2-normalized vector obtained by weighting and aggregating the previous view's content features using the transport matrix. It encodes "which content features in the previous frame were matched to style feature \(j\)." The role of the guidance map is: if a content feature in the current view is similar to \(g_j^{(v)}\) in the guidance map space, it indicates that it also "resembles" the content features matched to \(j\) in the previous frame, and thus should also be matched to \(j\). This achieves soft propagation of matching preferences across views.

Note: The guidance map only affects the construction of the KNN graph (determining \(E\)) and has no impact on the transport cost \(C\), which is always calculated using the pure content-style cosine distance. This ensures that style matching quality is not compromised by the guidance intensity. \(\lambda_{\mathit{guide}}=1\) achieves the optimal balance between consistency and content fidelity.

A Complete Example

Suppose a scene has three adjacent training views, \(v_1, v_2, v_3\). At \(v_1\), rendering features \(X^{(1)}\) (\(N=1024\) spatial positions) and style features \(Y\) (\(M=512\)) are extracted from the VGG relu3_1 layer. For each \(x_i^{(1)}\), a \(K=16\) nearest neighbor search is performed to construct the sparse edge set \(E\), and 5 Sinkhorn iterations are run to obtain the transport matrix \(T^{(1)}\). The CCFT loss is calculated as \(\mathcal{L}_{\mathit{CCFT}} = \sum T_{ij} C_{ij}\). Subsequently, the guidance map \(G^{(2)}\) is computed: for each style feature \(j\), \(g_j^{(2)}\in\mathbb{R}^C\) is the weighted sum (after L2 normalization) of all content features matched to \(j\) in \(v_1\). Upon moving to \(v_2\), the KNN search no longer relies solely on the similarity between \(x_i^{(2)}\) and \(y_j\) but instead uses the guided distance. If \(x_i^{(2)}\) corresponds to the region matched to the "sea wave texture" style feature in \(v_1\), it will be close to \(g_{\text{wave}}^{(2)}\) in the guidance map space, thus automatically guiding it to the same style feature. The same process applies to \(v_3\). After optimization with 50 iterations per view (for forward-facing scenes), all views eventually converge to a consistent and diverse style assignment.

Loss & Training

The total loss during the stylization stage is \(\mathcal{L}_{\mathit{style}} = \lambda_{\mathit{CCFT}}\mathcal{L}_{\mathit{CCFT}} + \lambda_{\mathit{content}}\mathcal{L}_{\mathit{content}}\), where \(\mathcal{L}_{\mathit{content}} = \frac{1}{N}\sum (x_i - \hat{x}_i)^2\) computes the MSE distance between the rendered features and the original scene features, with \(\lambda_{\mathit{CCFT}}=30\) and \(\lambda_{\mathit{content}}=0.005\). A critical training strategy choice: during the stylization stage, all geometric parameters (position, covariance, opacity) are frozen, and only the spherical harmonics coefficients (color parameters) are optimized. Optimization comparisons show that optimizing all parameters causes the structure loss to skyrocket from 0.0318 to 0.0607 (the scene geometry is forcibly distorted by the style loss), whereas optimizing only colors perfectly preserves the original 3D geometric structure while transferring the style. Additionally, a post-color transfer operation (following the practice of ARF) is applied before and after stylization to maintain color tone consistency with the reference image. For forward-facing scenes, 50 iterations per view are performed (total iterations \(\ge 2000\)), and 15-20 iterations per view for 360-degree scenes. Using a single RTX 4090 GPU, stylization takes an average of 8.35 minutes.

Key Experimental Results

Main Results

On the LLFF (forward-facing), T&T, and Mip-NeRF 360 (360-degree) datasets, 5 scenes \(\times\) 6 style reference images = 60 test cases. Compared methods: StylizedGS, ABC-GS, SGSST, CLIPGaussian, StyleGaussian.

Method	ArtFID↓	Structure Loss↓	Short MEt3R↓	Long MEt3R↓
StylizedGS	23.068	0.0462	0.1201	0.2858
ABC-GS	25.026	0.0357	0.1198	0.2822
SGSST	25.143	0.0568	0.1384	0.3064
CLIPGaussian	27.293	0.0431	0.1353	0.2903
StyleGaussian	31.855	0.0425	0.1284	0.2982
Ours (3DGS)	24.133	0.0320	0.1198	0.2795
Ours	22.801	0.0318	0.1196	0.2795

User study (25 participants, each evaluating 6 scenes, 450 total rankings): Our method is ranked first by 32.22% for style quality (SGSST is second with 25.04%), 37.61% for content fidelity (ABC-GS is second with 19.04%), and 37.61% for multi-view consistency (ABC-GS is second with 19.83%), dramatically leading in all three dimensions.

Ablation Study

Configuration	ArtFID↓	Structure Loss↓	Short MEt3R↓	Long MEt3R↓	Note
Full model (\(\lambda_{\mathit{guide}}=1\))	22.801	0.0318	0.1196	0.2795	Full model
w/o Cross-view guidance (\(\lambda_{\mathit{guide}}=0\))	22.809	0.0312	0.1204	0.2807	Short- and long-term consistency degrade without guidance
w/o Enhanced reconstruction (using vanilla 3DGS)	24.133	0.0320	0.1198	0.2795	ArtFID degrades significantly (24.133 vs 22.801)
Optimize all parameters (not only colors)	23.215	0.0607	0.1358	0.2905	Structure loss explodes, multi-view consistency collapses
Unordered view training	22.806	0.0315	0.1202	0.2804	Guidance effect is significantly weakened, approaching the unguided version
Increase guidance strength (\(\lambda_{\mathit{guide}}=10\))	22.805	0.0323	0.1191	0.2786	Consistency continues to improve but content structure slightly degrades

Progressive ablation of geometric regularization (visualized): When using only \(\mathcal{L}_{\mathit{photo}}\), scene geometry is disorganized; adding \(\mathcal{L}_{\mathit{depth}}\) stabilizes the global structure; adding \(\mathcal{L}_{\mathit{shape}}\) suppresses needle-like distorted Gaussians; and adding \(\mathcal{L}_{\mathit{size}}\) refines local details. Overlaying these four elements transforms the Gaussian primitives from messy and chaotic into uniform and dense, providing a more stable geometric baseline for stylization.

Loss function ablation: Replacing CCFT with NNFM Loss results in a monotonous sea wave texture in the background with repetitive patterns over large areas. Replacing it with FAST Loss captures some style patterns but lacks details and shows artifacts. Replacing it with GRAM Loss fails to transfer local style patterns and generates severe artifacts. CCFT remains the only solution that balances style details and distribution uniformity.

Key Findings

• CCFT loss is the core contribution: Column capacity constraints mathematically guarantee the suppression of many-to-one matching. The parameter \(\tau\) offers a continuous and controllable spectrum ranging from "greedy matching" to "uniform distribution," which NNFM, FAST, and GRAM fail to achieve.
• Tension exists between 2D and 3D in cross-view guidance: Increasing \(\lambda_{\mathit{guide}}\) improves consistency (Long MEt3R drops from 0.2807 to 0.2786) but slightly compromises the content structure (Structure Loss rises from 0.0312 to 0.0323). This is fundamentally because 2D style patterns override part of the 3D content boundaries, presenting a parameter trade-off that needs careful balance in practical scenarios.
• Enhanced reconstruction is a vital supporting contribution: Using the original 3DGS yields an ArtFID of 24.133, which improves to 22.801 when enhanced reconstruction is applied, representing a notable gap that proves geometric quality directly determines the upper bound of stylization.
• Color-only optimization is the most underrated design choice: Optimizing all parameters leads to a structure loss of 0.0607, while color optimization yields only 0.0318. The near-twofold difference proves that freezing geometry during the stylization stage is indispensable.

Highlights & Insights

• Replacing Nearest Neighbor Matching with Optimal Transport: Elevating style matching from a "non-differentiable, non-global" greedy search to a "differentiable, globally constrained" transport optimization is a conceptual vertical upgrade. Column capacity constraints explicitly model the many-to-one problem directly, yielding a mathematically elegant theory.
• Guidance Map Propagation as a Reusable Design Pattern: The core of cross-view guidance lies in "encoding the previous frame's optimal transport plan into a guidance map and injecting it into the next frame's KNN search." This paradigm of "softly guiding the current search space using historical optimal solutions" is not only suitable across views but can also be transferred to video-frame consistency and multi-modal feature alignment.
• Sparsified Sinkhorn Strategy Lowering Memory Costs: Keeping only the Top-K nearest neighbor edges on the bipartite graph (as opposed to full connection) reduces the OT computational complexity from \(O(NM)\) to \(O(NK)\). This practical engineering optimization maintains transfer quality while capping peak GPU memory usage at 8.37GB (lower than ABC-GS's 9.63GB).
• Decoupled Two-Stage Approach of Geometric Regularization and Stylization: First optimizing the reconstruction with a focus on geometry and then freezing the geometry to only optimize colors for stylization avoids "style loss pollution" on the geometry. This decoupled thinking is simple yet highly effective and can be readily extended to other 3D editing tasks.

Limitations & Future Work

• Intrinsic Contradiction in the 2D-3D Tension of Cross-View Guidance: Guidance naturally enforces consistency in the 2D rendering domain, but the scene itself is 3D. When 2D style patterns are excessively aligned, 3D content boundaries are erased. The authors acknowledge this trade-off and adopt \(\lambda_{\mathit{guide}}=1\) as a compromise, yet they do not offer an adaptive adjustment mechanism. A possible direction for improvement is using depth information to weight the guidance intensity—reducing \(\lambda_{\mathit{guide}}\) near geometric boundaries and increasing it in flat areas.
• Scene-by-Scene Optimization Lacks Real-Time Capability: Stylization takes around 8.35 minutes per scene, which is far from real-time or interactive applications. While the authors suggest feed-forward methods for future research, how to achieve feed-forward execution while preserving capacity constraints and cross-view guidance remains an open question.
• Dependence on View Order: The guidance mechanism assumes that training views are sequentially continuous in space (which is naturally met in forward-facing and 360-degree datasets). Randomized view orders significantly weaken the guidance effect (Long MEt3R falls back to 0.2804). For custom datasets with sparse or non-uniform camera trajectories, camera interpolation or inserting virtual views is required to restore continuity.
• Only Tested with VGG Features: All experiments rely on the relu3_1 layer of VGG-16, without exploring more modern feature extractors like DINO or CLIP. The impact of different backbone feature spaces on OT matching behaviors and guidance map quality is a dimension yet to be studied.
• Single Reference Image for Style: The method has not been tested with multi-style references or text-guided scene stylization.

Related Work & Insights

• vs ARF (NNFM): ARF introduced NNFM loss to pioneer high-quality 3D stylization, but its many-to-one nature leads to monotonous, repetitive textures. This paper directly replaces NNFM with CCFT to fundamentally solve this issue via column capacity constraints. Note that ARF is a special case of CCFT when \(\tau\to 0\).
• vs ABC-GS (FAST Loss): ABC-GS replaces NNFM with feature alignment to alleviate many-to-one matching, but the global linear transformation causes incorrect matching in regions with insufficient local semantics (e.g., matching sunflower seed patterns to a triceratops snout) and over-smoothes high-frequency features. In contrast, the proposed OT-based approach maintains local semantic precision while delivering explicit capacity control.
• vs StylizedGS / MultiStyleGS: The former uses depth preservation losses to enhance geometric consistency, and the latter adopts DINO features to improve feature uniqueness—both being indirect methods. The proposed cross-view guidance works directly at the matching mechanism level, marking the first work to explicitly constrain the same 3D content to match consistent style patterns across views.
• vs SGSST (Gram Loss): The Gram matrix, which is based on global statistics, causes structural collapse when there are major discrepancies between local content and style structures (e.g., making petals and leaves indistinguishable). The local OT matching proposed here naturally preserves space correspondences.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introducing optimal transport to 3DGS stylization with the combination of semi-balanced OT column capacity constraints and cross-view guidance maps shows genuine originality in problem modeling rather than a simple piling of modules.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ The main experiments cover 5 baselines, 3 datasets, and 60 test cases. The ablation study spans 6 dimensions (loss function, capacity control, guidance strength, geometric regularization, parameter optimization strategies, view sequencing). The user study involves 25 participants and 450 total rankings. The only missing part is an analysis of different VGG layers and backbones.
• Writing Quality: ⭐⭐⭐⭐☆ The problem description is clear, the motivation chain is complete (many-to-one + cross-view inconsistency \(\to\) two root causes \(\to\) two corresponding mechanism solutions), the methodological derivation is solid, and the appendix contains complete algorithm pseudocode. The sole minor drawback is that the OT theoretical foundation is somewhat lengthy and could be integrated more tightly with the core contributions.
• Value: ⭐⭐⭐⭐⭐ Multi-view consistency in 3DGS stylization is a widely recognized focal pain point in the field. This work is the first to provide a systematic solution at the matching mechanism level. The trio of CCFT loss, cross-view guidance, and enhanced reconstruction offers clear reference value and transfer potential for future work.

Related Papers

• [ICLR 2026] Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting
• [ECCV 2026] Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures
• [ECCV 2026] ViewSplat: View-Adaptive 3D Gaussian Splatting for Feed-Forward Synthesis
• [ECCV 2026] Geometry-Aware Style Transfer in 3D Gaussian Splatting
• [ECCV 2026] Monte Carlo Energy Aggregation for Mobile 3D Gaussian Splatting