StyleFusion360: View-Consistent Head Stylization via Adaptive Style Modulation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/furkanguzelant/StyleFusion360
Area: 3D Vision
Keywords: 3D Head Stylization, Diffusion Models, Style Fusion Attention, Multi-view Consistency, Adaptive Key Modulation

TL;DR¶

Addressing the challenge of balancing fine geometric preservation with zero-shot generalization across unseen styles, StyleFusion360 builds upon a 3D-aware diffusion backbone with decoupled appearance branches and introduces a two-stage Style Fusion Attention with adaptive key modulation, delivering 360-degree view-consistent and identity-preserving head stylization without per-style retraining.

Background & Motivation¶

3D head stylization aims to transform realistic human faces into expressive, varied artistic representations, holding tremendous utility in digital character creation, gaming animation, and immersive media. However, transferring artistic styles while faithfully preserving subject identity and guaranteeing 360-degree multi-view geometric consistency remains a formidable challenge.

Existing approaches predominantly operate in the 2D domain. While single-image 2D stylization exhibits impressive visual creativity, applying it independently across views inevitably induces severe flickering and geometric drift. A naive remedy—first stylizing a 2D portrait and subsequently reconstructing it into 3D using single-view reconstruction backbones—frequently falters because reconstruction models assume photorealistic human facial priors; the non-photorealistic artistic distortions disrupt structural cues, resulting in geometric collapse, identity loss, and degraded textures. Conversely, 3D GAN-based methods, bounded by compact tri-plane or volumetric representations, face significant bottlenecks in texture realism, back-of-head synthesis, and rendering intricate accessories such as glasses or hats. Furthermore, most 3D GAN pipelines require time-consuming per-style optimization or domain fine-tuning. While recent 3D-aware diffusion frameworks provide rich structural priors, effectively injecting arbitrary style references into a multi-view consistent latent space without disrupting facial geometry remains an open problem.

The primary tension lies in the fact that keeping the structural pathway frozen is indispensable for preserving delicate facial geometry, yet this frozen pathway easily dominates and repels incoming style signals; simultaneously, naive global modulations fail to capture spatially unaligned, view-varying semantic correspondences between reference style and source face. To overcome this limitation, this paper explores feature-level non-rigid spatial alignment directly within a 3D-aware diffusion framework. Core Idea: Decouple identity and style appearance extraction into separate modules, and incorporate a two-stage Style Fusion Attention mechanism that performs cross-attention key modulation and AdaIN feature statistics injection on content keys, achieving high-fidelity, controllable, and multi-view consistent 3D head stylization without per-style retraining.

Method¶

Overall Architecture¶

StyleFusion360 is built upon a 3D-aware latent diffusion backbone for multi-view portrait synthesis (DiffPortrait360). The overall system takes three inputs: a single content face image $I_{\text{content}}$, an arbitrary reference style image $I_{\text{style}}$, and a target camera pose parameter $I_{\text{cam}}$. A frozen Content Appearance Module extracts structural and identity embeddings, while a trainable Style Appearance Module extracts artistic textures and color patterns. In the primary generation diffusion U-Net, the proposed Style Fusion Attention fuses these features to synthesize view-consistent stylized portraits across full 360-degree perspectives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Content Image + Style Image + Camera Pose"] --> B["Dual Appearance Feature Extraction<br/>Frozen Content Branch / Trainable Style Branch"]
    B --> C["Content Key Modulation<br/>Key Cross-Attention + AdaIN Statistics Injection"]
    C --> D["Shared Cross-Attention<br/>Diffusion Latent Feature Querying + Multi-view Fusion"]
    D --> E["Stylization Intensity Slider & Local Mask Editing<br/>Temperature Scaling + Region-Aware Style Mixing"]
    E --> F["Output: 360° View-Consistent Stylized Head"]

Key Designs¶

1. Decoupled Dual Appearance Modules: Isolating Geometry from Artistic Textures To prevent artistic style injection from degrading delicate facial geometry, the architecture decouples appearance feature extraction into two independent branches. The Content Appearance Module, initialized from the 3D-aware diffusion backbone, is strictly frozen during training. This retains the rich geometric and identity priors learned from high-quality real faces, preserving fine-grained structures like facial contours, hair strands, glasses, and headwear. In contrast, the Style Appearance Module is initialized from the content encoder but fine-tuned to extract style-specific representations (e.g., color distribution, stroke patterns, and material shading) from $I_{\text{style}}$, fundamentally separating geometric identity from artistic attributes.

2. Content Key Modulation: Discovering Soft Correspondences and Adaptive Normalization Directly mixing style key-values into diffusion cross-attention fails because the frozen content pathway tends to dominate the generation, and the content and style images are typically unaligned with differing camera poses. Uniform global modulation (such as StyleAligned) cannot handle spatially varying facial regions (e.g., eyes, hair, skin). The authors design a dedicated two-stage attention mechanism. In Stage 1, content key features $K_c$ act as queries, while linearly projected style key features $K_s^m$ and $V_s^m$ act as keys and values in a secondary cross-attention layer: $$F_{cs} = \text{Softmax}\left(\frac{Q_c^m (K_s^m)^T}{\sqrt{d}}\right) V_s^m$$ This step discovers soft spatial correspondences between content and style across disparate poses. Next, Adaptive Instance Normalization (AdaIN) transfers the extracted style statistics into the original content key: $\hat{K}_c = \text{AdaIN}(K_c, F_{cs})$. This adaptive modulation prevents the frozen content path from suppressing style signals while achieving region-aware style transfer.

3. Shared Cross-Attention and Multi-View Diffusion Training In the final attention layers of the diffusion U-Net backbone (Stage 2), latent diffusion features $Q_l$ serve as queries, attending jointly across modulated content keys and raw style keys: $$K = [\hat{K}_c, K_s], \quad V = [V_c, V_s]$$ $$A = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d}}\right) V$$ Retaining raw style representations in $K$ and $V$ provides direct artistic guidance alongside the modulated content features. For training, paired multi-view data synthesized via a stylized 3D GAN (PanoHead adapted on a few base styles) is used to fine-tune the Style Appearance Module, Style Fusion Attention, and View Consistency Module. Although synthetic 3D GAN images suffer from limited realism, the frozen Content Appearance Module grounds the generator in real human priors, allowing StyleFusion360 to generalize zero-shot to real photos and unseen styles in approximately 3 minutes at inference without per-style retraining.

4. Continuous Intensity Slider and Local Multi-Reference Editing To support flexible user interaction, the method incorporates temperature scaling on the style keys $K_s$ within Style Fusion Attention. By dividing the style projection by a user-tunable temperature scalar $\tau > 1$ before Softmax, users can continuously adjust stylization strength from subtle, structure-dominant touch-ups to vivid, heavy stylization. Furthermore, combined with binary masks $M$ from off-the-shelf segmentation models (e.g., SAM 2), the system supports localized region-specific edits (e.g., applying a Pixar style to the eyes while applying a Joker style to the mouth) and linear feature interpolation between multiple style references in a single forward pass.

Key Experimental Results¶

Main Results¶

Evaluation was conducted on FFHQ and RenderMe360 (providing 360-degree ground-truth multi-view images and depth). Baselines include state-of-the-art 2D stylization + novel-view pipelines (IP2P, InstantID, StyleCLIP, OmniStyle, StyleID, StyleShot) and 3D GAN-based stylization methods (StyleGAN-Fusion, StyleGAN-NADA, DiffusionGAN3D, Identity3DHead).

Dataset	Metric	StyleFusion360 (Ours)	Prev. SOTA / Best Baseline	Gain / Difference
FFHQ	FID ↓	64.48	86.23 (StyleShot) / 88.26 (Identity3DHead)	-21.75
FFHQ	CLIP ↑	0.8237	0.8209 (Identity3DHead)	+0.0028
FFHQ	MEt3R ↓	9.99	9.68 (DiffusionGAN3D) / 10.76 (StyleCLIP)	Competitive with explicit 3D
FFHQ	User Study ↑	76.0%	14.0% (Identity3DHead)	Overwhelming preference
RenderMe360	FID ↓	74.86	90.58 (StyleID) / 96.95 (StyleGAN-Fusion)	-15.72
RenderMe360	CLIP ↑	0.7939	0.7279 (StyleShot) / 0.7155 (Identity3DHead)	+0.0660
RenderMe360	$\Delta D$ ↓	0.0235	0.0303 (StyleGAN-Fusion) / 0.0307 (DiffusionGAN3D)	22.4% lower depth error
RenderMe360	ID Score ↑	0.45	0.43 (StyleID) / 0.36 (DiffusionGAN3D)	Superior identity preservation
RenderMe360	MEt3R ↓	15.55	15.20 (DiffusionGAN3D) / 16.44 (Identity3DHead)	High cross-view consistency

Note: $\Delta D$ represents the L2 depth difference between generated images and unedited multi-view ground truth (estimated via Depth Anything v2); MEt3R measures loop-closed consecutive view consistency; User Study represents blind perceptual preference against the top three baselines.

Ablation Study¶

The ablation investigates the core architectural choices in content key modulation and compares them against alternative style transfer techniques.

Config	Performance & Visual Behavior	Note
Full Model	Optimal balance of expressive stylization, identity preservation, and view consistency	Best overall configuration
w/o AdaIN Modulation	Stylization severely inhibited; outputs stay overly close to input content	Confirms AdaIN is essential for injecting global style statistics
w/o Key Attention	Incoherent spatial stylization; transfers color patches without semantic alignment	Lack of cross-attention leads to misaligned feature injection
Replaced with StyleAligned	Produces view-dependent artifacts and breaks multi-view consistency	Uniform feature matching fails in complex 3D facial geometry
w/ Value Feature Modulation	Results nearly identical to Key-only modulation while increasing computation	Key modulation alone is sufficient; Value modulation omitted

Key Findings¶

3D Diffusion Outperforms Cascade Pipelines: 2D stylization coupled with DiffPortrait360 yields notably inferior FID and CLIP scores, demonstrating that 2D artistic transformations distort structural cues required by novel-view synthesizers.
Superior Geometry over Explicit 3D GANs: On RenderMe360, StyleFusion360 achieves a lower depth difference $\Delta D$ (0.0235) than tri-plane GAN baselines (0.0303~0.0365), confirming that implicit diffusion priors maintain more accurate and stable 3D facial geometry.
Expression Disentanglement Robustness: Intentionally altering the facial expression of the reference style image (e.g., from neutral to laughing or angry) leaves the generated portrait's facial geometry and expression virtually unchanged, demonstrating that the key modulation extracts high-level stylistic motifs rather than transient expression geometry.

Highlights & Insights¶

Key Cross-Attention for Soft Correspondence: Formulating key modulation as a cross-attention step between content and style embeddings elegantly resolves the spatial and pose misalignment between reference style and subject portraits.
Overcoming Synthetic Data Degradation via Frozen Priors: Training on coarse 3D GAN-rendered multi-view pairs while freezing the content pathway allows the model to learn cross-view coherence without inheriting the synthetic artifacts of the training data.
Inference-Time Temperature Controllability: Using a simple temperature scaling factor on style keys enables smooth, continuous control over style intensity without requiring model retraining or latent space optimization.

Limitations & Future Work¶

Extreme Morphological Deformations: For styles requiring non-human cranial anatomy or radical structural distortions (e.g., exaggerated cartoon skull structures), balancing identity preservation with severe geometric deformation remains challenging.
Dynamic Expression Integration: The current pipeline focuses on static 3D heads; integrating motion-conditioned video diffusion models (such as Zero-1-to-A or SOAP) could enable interactive, animated 3D stylized avatars with talking capabilities.

vs. Identity3DHead & DiffusionGAN3D: GAN-based 3D stylization requires per-style optimization or hours of generator retraining, struggling with back-of-head textures and accessories; StyleFusion360 operates in a feed-forward manner (~3 mins per new style) while faithfully rendering hats, glasses, and full 360-degree views.
vs. StyleAligned & 2D Diffusion Baselines: StyleAligned imposes global feature statistics uniformly across spatial tokens, causing view inconsistency in 3D; StyleFusion360 establishes local, spatially adaptive soft correspondences through key cross-attention, ensuring view-consistent style transfer.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Elegantly adapts key-space cross-attention and AdaIN within a 3D-aware diffusion backbone, solving the style-geometry tension.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive comparisons across 10+ 2D/3D baselines on FFHQ and RenderMe360, with rigorous depth, consistency, and perceptual evaluations.
Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, well-structured methodology, and insightful ablation analysis.
Value: ⭐⭐⭐⭐⭐ Highly practical framework for zero-shot 3D avatar creation, gaming, and digital human animation.