COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing¶

Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Compositional GAN, Disentangled Editing, Human Head Synthesis, Unsupervised Semantic Decomposition

TL;DR¶

COSY presents a compositional 3D Gaussian Splatting GAN architecture that decomposes human head generation into four independent sub-generators (hair, skin, glasses, and torso) guided by sparse color histograms and boolean flags without explicit geometric templates, enabling real-time, zero-leakage 3D attribute editing.

Background & Motivation¶

Synthesizing photorealistic, controllable 3D human heads is essential for virtual reality, film production, interactive gaming, and synthetic data generation for computer vision. These downstream applications require users to intuitively manipulate specific semantic attributes—such as hairstyles, accessories, or facial features—in real-time while strictly preserving the underlying identity, head geometry, and physical coherence. 3D-aware Generative Adversarial Networks (3D GANs), transitioning from implicit neural radiance fields to explicit 3D Gaussian Splatting (3DGS) representations, have emerged as a dominant paradigm because they learn explicit 3D structures directly from unstructured in-the-wild 2D photograph collections, provide real-time rasterization speeds, and maintain smooth latent manifolds suitable for GAN inversion and interactive manipulation.

However, the core hurdle of existing 3D GANs lies in their unconditionally entangled global latent space. Because real-world human head datasets naturally exhibit strong statistical correlations and biases (e.g., hair color strongly correlates with skin tone; specific hairstyles co-occur with accessories), edits performed within an entangled latent space inevitably trigger unintended global side effects. Post-training latent manipulation techniques—such as PCA-based trajectory discovery (GANSpace) or classifier-guided hyperplanes (InterFaceGAN, StyleFlow)—struggle with feature leakage, causing unintended shifts in facial identity or hair geometry when merely adding glasses. Conversely, reference-based feature blending or 2D segmentation-guided generators either require minutes per frame, breaking real-time interactivity, or rely heavily on strict geometric priors such as FLAME head meshes or dense 2D segmentation supervision, which severely constrains non-mesh topologies like curly hair and diverse eyeglasses.

This paper breaks away from monolithic generator designs by decomposing 3D head synthesis into semantically localized, independent sub-generators guided solely by sparse conditionings. Core idea: decompose 3D human head synthesis into independent 3DGS sub-generators for hair, skin, glasses, and torso without explicit segmentation masks or geometric priors, coordinating global geometric and lighting consistency via minimal shared context tokens and training-time latent mixing to achieve strictly disentangled, real-time 3D head editing.

Method¶

Overall Architecture¶

COSY builds upon unconstrained point-cloud Gaussian generation (CGS-GAN) without imposing predefined template meshes. The pipeline operates in four coordinated stages: first, a shared transformer backbone processes the input latent code and splits latent representations among four dedicated sub-generators responsible for hair, skin/face, glasses, and torso. Second, each sub-generator independently synthesizes its localized 3D Gaussian primitives. Third, sparse conditioning signals—a boolean presence flag for glasses and sparse 10-bin RGB color histograms for permanent parts—are injected at the final decoder layers to implicitly steer semantic specialization. Fourth, all Gaussian primitives from the active sub-generators are concatenated into a unified 3D scene, rendered into 2D images via differentiable 3DGS rasterization, and evaluated by an image-level discriminator. Global geometric cohesion and illumination alignment are maintained through training-time latent mixing, a shared shape context token, a light context token, and an exponential moving average (EMA) face geometry prior.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Latents z<br/>+ Shared Contexts (wShape, wLight)"] --> Partition["Backbone Feature Decomposition<br/>Dispatched to 4 semantic sub-generators"]
    Partition --> SubGens["Independent 3DGS Sub-Generators<br/>Synthesize unconstrained Gaussian primitives"]
    SubGens --> SparseMod["Sparse Conditioning Decoder Layers<br/>Color histogram modulation + opacity switch"]
    SparseMod --> Mixing["Training-Time Latent Mixing<br/>Randomly swap sub-features across identities"]
    Mixing --> GlobalReg["Dual Context Regulation & EMA Face Prior<br/>Align global silhouette, lighting, and forehead"]
    GlobalReg --> SplatCompose["Full 3DGS Primitive Composition<br/>Differentiable splatting & image-level discrimination"]

Key Designs¶

1. Sparse Attribute-Driven Compositional 3DGS Architecture: Template-Free Semantic Disentanglement

Prior compositional models strictly require dense 2D segmentation masks or parametric FLAME meshes, limiting their expressiveness on unconstrained hairstyles and accessories. COSY allocates an adapted CGS-GAN generator branch to each semantic component and introduces semantic specialization through sparse supervision. For additive components that are optionally present (e.g., glasses), the sub-generator receives a binary indicator \(b \in \{0, 1\}\); when \(b=0\), the generator deducts a large offset from the Gaussian opacity attributes, completely hiding the component. If the glasses branch attempts to synthesize parts of the eyes or face, those regions disappear from the full composition when \(b=0\), incurring an adversarial penalty from the discriminator and forcing the branch to self-restrict strictly to glasses. For permanent components (skin, hair, and torso), COSY extracts a sparse 10-bin RGB histogram over coarse facial parsing regions as an attribute condition. This histogram conditions only the final color-modulation decoder layer of the respective sub-generator without affecting point positions. Consequently, the network learns an explicit mapping from color inputs to Gaussian color attributes, maintaining robust geometry even when queried with out-of-distribution, novel color combinations during inference.

2. Training-Time Latent Mixing: Enforcing Boundary Compatibility Across Identity Combinations

Synthesizing individual components from independent latent vectors during inference can cause visible spatial seams, gaps, or unnatural overlaps at component boundaries if the sub-generators are trained in isolation. To bridge this gap, COSY introduces latent mixing directly into the training loop: during each forward pass, the backbone feature of one semantic component is replaced with high probability (10% by default) by that of another random identity from the batch. The resulting hybridized 3DGS scene is rendered as a whole and fed to the unmasked image-level discriminator. The discriminator penalizes seams, floating artifacts, or unnatural transitions at boundaries between mismatched hair and face geometries. This process conditions each sub-generator to produce components with clean, universally compatible interface margins.

3. Minimal Shared Context Tokens: Balancing Global Consistency Against Attribute Leakage

While latent mixing encourages cross-component compatibility, components from disparate training subjects can possess mismatched global head proportions or contradictory directional illumination. Providing unrestricted global feature sharing would reintroduce semantic entanglement, causing hair edits to alter face identity. COSY resolves this tension via two minimal, restricted context tokens: \(w_{\text{Shape}} \in \mathbb{R}^{512}\) and \(w_{\text{Light}} \in \mathbb{R}^{512}\), which are excluded from latent mixing. The shape context \(w_{\text{Shape}}\) is routed exclusively to the final xyz coordinate decoder MLP of each sub-generator. Because this lightweight MLP possesses minimal expressive capacity, it can only rescale and adjust global boundaries without leaking identity-level facial textures. For illumination, the light context \(w_{\text{Light}}\) modulates the generator inputs to accommodate complex cast shadows and specular highlights; to prevent it from altering geometry or identity, an LPIPS perceptual regularization constraint is enforced across pairs rendered under different light contexts, restricting variations strictly to lighting and shading. Furthermore, an Exponential Moving Average (EMA) face geometry regularization is applied to prevent the face branch from skipping covered regions (such as the upper forehead under bangs), ensuring full geometry completion when hairstyles are swapped.

Loss & Training¶

The overall model is trained using non-saturating adversarial loss with \(R_1\) gradient regularization. The conditional discriminator receives full rendered images alongside concatenated camera parameters, color histograms, and the boolean glasses indicator. Training incorporates two specialized auxiliary regularizations: an illumination perceptual loss \(\mathcal{L}_{\text{light}} = \text{LPIPS}(I_{w_{\text{Light}}^1}, I_{w_{\text{Light}}^2})\) calibrated to a target mean distance of 0.3, which restricts \(w_{\text{Light}}\) updates to lighting changes, and an EMA face geometry loss \(\mathcal{L}_{\text{EMA}}\) penalizing deviations of Gaussian coordinates and opacities from a running average face geometry. Training is conducted on the FFHQC dataset at 512×512 and 1024×1024 resolutions with an optimal latent mixing probability of 10%.

Key Experimental Results¶

Main Results¶

The method is evaluated against recent unconditional and conditional 3DGS GAN baselines on FFHQC using the view-unconditioned \(\text{FID3D}\) metric. Disentangled editing precision is assessed by adding glasses or altering hair on 1,000 baseline subjects and computing the \(\text{LPIPS}\) shift in unaffected regions (masked via BiSeNet) alongside the empirical recall of successfully synthesized target attributes.

Table 1: Unconditional 3D Image Quality (FID3D) on FFHQC at 512² and 1024² Resolutions (Reflecting original Table 1)

Method	Representation Type	FID3D ↓ (512×512)	FID3D ↓ (1024×1024)
GSGAN	Hierarchical Point Cloud 3DGS	7.68	-
GGHead	FLAME Template Mesh 3DGS	7.78	14.27
CGS-GAN	Multi-view Consistent Point Cloud 3DGS	4.53	5.25
Cond CGS-GAN	Naive Conditional Point Cloud 3DGS	4.52	-
COSY (Ours)	Compositional Multi-branch 3DGS	4.42	5.43

Note: EGG3D synthesizes background (FID 7.51 on FFHQ-512) and is omitted from direct foreground-only comparison due to background metric sensitivity.

Table 2: Editing Disentanglement (LPIPS in Unrelated Regions) and Glasses Recall (Consolidated from original Tables 2 & 3)

Editing Method	Editing Paradigm	Glasses Edit Unrelated LPIPS ↓	Hair Edit Unrelated LPIPS ↓	Target Glasses Recall ↑
GANSpace	Unsupervised Latent PCA Traversal	0.254	0.165	50.8% (0.508)
InterFaceGAN	Pre-trained Classifier Hyperplane	0.196	-	63.2% (0.632)
StyleFlow	Continuous Normalizing Flow Mapping	0.172	0.133	94.8% (0.948)
Cond CGS-GAN	Monolithic Conditional 3D GAN	0.239	0.149	95.2% (0.952)
COSY (Ours)	Compositional Independent Branches	0.016	0.014	99.7% (0.997)

Ablation Study¶

The impact of training-time latent mixing probabilities on single-identity generation quality (\(\text{FID}\)) versus cross-identity compositional quality (\(\text{FIDMix}\)) demonstrates the operational trade-off.

Table 3: Ablation of Latent Mixing Probability on FID and FIDMix (Reflecting original Table 4)

Mixing Probability	Image Quality FID ↓	Composition Quality FIDMix ↓	Observation
0% (No mixing during training)	4.88	10.51	Severe boundary seams and missing geometry occur during inference composition
10% (COSY Default)	4.42	8.32	Optimal sweet spot balancing individual realism and seamless cross-identity blending
20%	4.98	9.07	Generator capacity diverted toward over-regularizing boundary transitions
30%	5.16	8.49	Monolithic FID degrades further
40%	5.97	6.53	Best composition score but severe degradation of standard visual realism (FID slips to 5.97)

Key Findings¶

Structural Independence Eliminates Attribute Leakage: As evidenced by Table 2, conventional latent manipulation methods incur substantial perceptual shifts in non-target regions (\(\text{LPIPS} \ge 0.133\)), causing visible facial distortion when toggling glasses or hair. COSY reduces unrelated drift to \(0.014 \sim 0.016\) (a minor shift corresponding almost entirely to subtle, photorealistic cast shadows under the frame), while achieving an outstanding attribute recall of 99.7%.
10% Mixing Probability Strikes the Global Balance: Training without latent mixing leaves the model vulnerable to catastrophic boundary mismatch (\(\text{FIDMix} = 10.51\)), whereas excessive mixing (\(40\%\)) degrades base rendering fidelity (\(\text{FID} = 5.97\)). A 10% rate allows the image-level discriminator to penalize interface seams without cannibalizing generation fidelity (\(\text{FID} = 4.42\)).
Restricted Token Routing Prevents Information Bypass: Injecting \(w_{\text{Shape}}\) exclusively into the terminal coordinate decoder prevents identity-defining facial traits from leaking into the global shape channel, enabling robust head proportion scaling while preserving facial recognizability.

Highlights & Insights¶

Sparse Weak Supervision for Unsupervised Structural Decomposition: Instead of relying on rigid 3D parametric morphable models (e.g., FLAME) or dense per-frame 2D semantic masks, COSY achieves clean 3D component separation through simple 10-bin color histograms and binary flags, avoiding annotation overhead and template bias.
Physical Operation Mapping via Native 3DGS Attributes: Accessories are toggled by directly suppressing Gaussian opacities, and hairstyles are swapped by interchanging localized latent tokens. This bypasses iterative diffusion inversion steps and enables native real-time 3D interactive editing.
Broad Extensibility Across Modular Domains: The design of "decoupled backbone features \(\rightarrow\) independent localized 3DGS sub-generators \(\rightarrow\) shared minimal context tokens \(\rightarrow\) unified rendering and discrimination" is inherently domain-agnostic, providing a blueprint for full-body avatars (virtual try-on with decoupled garment geometry) or modular vehicle synthesis.

Limitations & Future Work¶

Handling Complex Component-to-Component Physical Contact: The independent sub-generators lack explicit physical collision handling, which may produce geometric interpenetration when modeling tight accessories (e.g., caps or helmets that physically deform the underlying hair) or dense facial beards.
Sensitivity of Illumination Regularization Weight: The light context regularization relies heavily on empirical tuning of the perceptual loss weight (calibrating mean LPIPS to \(\approx 0.3\)). Improper weighting causes either negligible lighting control or inadvertent facial texture drift, suggesting that future iterations would benefit from explicit spherical harmonics decomposition or neural environment mapping.

vs CGS-GAN / GSGAN: Both pioneer unconstrained point-cloud 3DGS GANs with high multi-view consistency, but operate in a monolithic latent space where attributes are entangled. COSY decomposes CGS-GAN into dedicated sub-generators, adding clean disentanglement without compromising multi-view fidelity.
vs EGG3D / GGHead: EGG3D and GGHead anchor Gaussians onto FLAME head meshes, which inherently limits their ability to model complex hair silhouettes and diverse glasses geometry. COSY dispenses with mesh templates entirely, achieving superior geometric freedom.
vs GANSpace / InterFaceGAN / StyleFlow: Latent vector manipulation methods struggle with dataset correlation biases, inevitably causing identity leakage during attribute edits. COSY resolves this structurally at the generator architecture level.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneers unconstrained compositional 3DGS generation guided solely by sparse color and binary conditions without 3D mesh priors or dense segmentation masks.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation covering FID3D, non-target LPIPS stability, attribute recall, and rigorous ablations on mixing rates and context tokens.]
Writing Quality: ⭐⭐⭐⭐⭐ [Clearly articulates the tension between monolithic entanglement and multi-component coordination, supported by clean pipeline visualizations and convincing ablations.]
Value: ⭐⭐⭐⭐⭐ [Delivers an efficient, real-time, highly disentangled framework for 3D avatar editing and scalable synthetic data generation.]