Skip to content

UniCSG: Unified High-Fidelity Content-Constrained Style-Driven Generation via Staged Semantic and Frequency Disentanglement

Conference: ECCV 2026
Paper: ECCV 2026 Poster 4742
Area: Image Generation
Keywords: style transfer, diffusion models, content-style disentanglement, multi-scale frequency decomposition, reinforcement learning alignment

TL;DR

UniCSG introduces a staged semantic and frequency disentanglement framework that builds a robust semantic scaffold via low-pass filtering and conditioning corruption in Stage 1, refines fine-grained textures via latent DoG pyramid supervision in Stage 2, and aligns perceptual quality via pixel-space rewards, substantially suppressing reference-content leakage in both text-guided and reference-guided style transfer.

Background & Motivation

Style transfer is a cornerstone task in controllable visual generation, encompassing both text-guided transfer (synthesizing artistic styles guided by natural language prompts) and reference-guided transfer (imparting the visual appearance of an exemplar image onto a target content image). Across artistic creation, digital advertising, and game asset development, the primary objective is to achieve expressive and faithful stylistic alignment while strictly preserving the geometric topology and semantic layout of the source content. However, as the foundational architecture of diffusion models has shifted from UNets with local inductive biases toward Diffusion Transformers (DiTs) with superior global scalability, models increasingly suffer from severe content–style entanglement. This entanglement frequently results in style overfitting, structural collapse, and reference-content leakage—where salient semantic entities from the style reference inadvertently overwrite the target content.

A deep investigation reveals three underlying bottlenecks responsible for these failures. First, latent diffusion models are typically optimized using a single, monolithic reconstruction objective that inherently prioritizes low-level textures and random pixel noise, under-emphasizing essential high-level semantic contours and structural composition. Second, frequency coupling exacerbates the mutual interference between content and style. DiT backbones naturally capture low-frequency global structures much more effectively than high-frequency textures, yet conventional frameworks fail to decouple these disparate frequency bands during training. Third, model optimization is conducted entirely within the latent space of a variational autoencoder (VAE), whereas human perceptual evaluation occurs in the decoded pixel space; solutions that achieve minimal mean squared error in latent space often diverge significantly from human perceptual optimality after decoding, introducing blurry artifacts or subtle structural drift.

To resolve these foundational tensions, this work departs from complex hybrid encoder designs or parameter-heavy workarounds, introducing a staged and progressive disentanglement paradigm. Core idea: decompose content-constrained style generation into staged latent-space low-frequency semantic disentanglement and multi-scale frequency detail reconstruction, coupled with pixel-space reward alignment, to realize unified, high-fidelity stylization under both text and reference guidance.

Method

Overall Architecture

UniCSG is built upon the pretrained Qwen-Image-Edit backbone and unifies text-to-image and image-to-image settings through a standardized four-tuple input \(\langle \text{text}, \mathbf{I}_{\text{ref}}, \mathbf{I}_{\text{content}}, \mathbf{I}_{\text{gt}} \rangle\). The training workflow executes a coordinated "disentangle–reconstruct–align" schedule across both latent and pixel spaces. In latent space, Stage 1 establishes a rigid semantic geometric blueprint by applying input low-pass filtering alongside conditioning corruption; Stage 2 removes the low-pass restriction and leverages a Difference of Gaussian (DoG) pyramid to supervise high-frequency texture recovery. Concurrently, a multi-dimensional pixel-space reward module decodes generated latents across both stages and applies policy-gradient optimization to align latent objectives directly with human perceptual quality.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Four-Tuple Data<br/>text / reference / content / target"] --> B["Stage 1: Low-Frequency Semantic Disentanglement<br/>Low-pass filtering + conditioning corruption + semantic contrastive loss"]
    B --> C["Stage 2: Frequency-Aware Detail Reconstruction<br/>Latent DoG pyramid decomposition + composite MSE/LogMSE loss"]
    C --> D["Pixel-Space Multi-Dimensional Reward Alignment<br/>CLIP-I content fidelity + CSD style alignment + policy gradient"]
    D --> E["High-Fidelity Stylized Output<br/>Rigid content topology preservation with zero reference leakage"]

Key Designs

1. Low-frequency semantic disentanglement: low-pass filtering and conditioning corruption To counter the tendency of diffusion models to rely on fine-grained textures early in training, Stage 1 exploits the natural coarse-to-fine trajectory of diffusion denoising. Before VAE encoding, low-pass filtering \(\mathcal{F}_{\text{low\_pass}}\) is applied to the target \(\mathbf{I}_{\text{target}}\), content \(\mathbf{I}_{\text{content}}\), and style \(\mathbf{I}_{\text{style}}\) images, retaining only spectral frequencies below threshold \(\tau\) (default \(\tau=0.2\)). To enforce explicit semantic separation under a unified timestep \(t\), the framework introduces differential noise amplification factors \(\gamma_{\text{style}} > \gamma_{\text{content}} > 1\). The augmented noise distribution for each conditioning branch is constructed as:

\[\boldsymbol{\epsilon}_{\gamma_i} = \boldsymbol{\epsilon} + \sqrt{\gamma_i^2 - 1}\,\boldsymbol{\epsilon}',\quad \boldsymbol{\epsilon},\boldsymbol{\epsilon}' \sim \mathcal{N}(0, \mathbf{I}) \implies \boldsymbol{\epsilon}_{\gamma_i} \sim \mathcal{N}(0, \gamma_i^2\mathbf{I})\]

This formulation creates a controlled information hierarchy under identical timestep scheduling: the target branch remains the least corrupted, the content branch is moderately degraded to preserve structural contours, and the style branch is subjected to the heaviest perturbation. This asymmetric degradation forces the DiT backbone to extract global stylistic characteristics exclusively from the severely corrupted style latent while deriving spatial layouts from the content latent, fundamentally eliminating reference-content leakage. Additionally, Stage 1 applies random replacement with pure noise at probability \(p\) to bolster robustness, optimized via base velocity prediction \(\mathcal{L}_{\text{base}}\) alongside a CLIP-guided semantic loss \(\mathcal{L}_{\text{sem}}\) composed of target fidelity \(\mathcal{L}_{\text{target}}^{\text{cont}}\), content preservation \(\mathcal{L}_{\text{content}}^{\text{cont}}\), and style repulsion \(\mathcal{L}_{\text{style}}^{\text{diss}}\).

2. Frequency-aware detail reconstruction: latent DoG pyramid and logarithmic high-frequency supervision Once the network has established a reliable semantic scaffold in Stage 1, Stage 2 lifts the low-pass filter to train across the full frequency spectrum. To ensure direct consistency with latent-space denoising objectives without intermediate feature decoding overhead, UniCSG computes multi-scale frequency decomposition directly in latent space using a Difference of Gaussian (DoG) pyramid, separating the predicted latent \(\hat{\mathbf{z}}\) and ground-truth latent \(\mathbf{z}_{\text{target}}\) into multi-scale low-frequency components \(\hat{\mathbf{z}}_{\text{low}}^{(k)}\) and high-frequency components \(\hat{\mathbf{z}}_{\text{high}}^{(k)}\).

Because low-frequency structures dictate global layout and exhibit uniform error characteristics, standard Mean Squared Error (MSE) provides strong, stable gradients. Conversely, high-frequency components contain a mixture of target stylistic textures and incidental noise; imposing strict pixel-wise L2 minimization frequently leads to severe overfitting and checkerboard artifacts. UniCSG addresses this by introducing Logarithmic MSE (LogMSE) on the high-frequency components, which naturally down-weights large outlying errors while remaining sensitive to small-to-moderate texture discrepancies:

\[\mathcal{L}_{\text{stage2}} = \mathcal{L}_{\text{base}} + \lambda_{\text{freq}} \sum_k \left[ w_{\text{low}} \left\| \hat{\mathbf{z}}_{\text{low}}^{(k)} - \mathbf{z}_{\text{target, low}}^{(k)} \right\|_2^2 + w_{\text{high}} \log\left( 1 + \left\| \hat{\mathbf{z}}_{\text{high}}^{(k)} - \mathbf{z}_{\text{target, high}}^{(k)} \right\|_2^2 \right) \right]\]

Setting \(w_{\text{high}} > w_{\text{low}}\) prioritizes fine-grained texture synthesis without disrupting the low-frequency geometric foundation established in Stage 1.

3. Pixel-space reward learning: cross-space perceptual consistency feedback Recognizing that latent-optimal solutions frequently diverge from human perceptual quality after decoding, UniCSG incorporates a continuous pixel-space reward learning mechanism throughout both training stages. Generated latents are decoded into pixel images \(\hat{\mathbf{I}}\) by the VAE decoder and evaluated by an ensemble of specialized perceptual scoring models:

\[R_{\text{total}} = \omega_c R_c + \omega_s R_s + \sum_i \omega_i R_i\]

Here, \(R_c\) measures semantic fidelity against the content image using CLIP-I, safeguarding primary object identity and layout; \(R_s\) evaluates stylistic congruence against the style reference using Cosine Style Distance (CSD); and auxiliary rewards \(R_i\) integrate LPIPS perceptual similarity and adversarial discriminator scores. Using a policy-gradient objective with step-wise advantage \(\mathcal{A}(s) = R_{\text{total}} - b(\cdot)\), the perceptual reward backpropagates directly into the diffusion backbone parameters, bridging the gap between latent reconstruction loss and visible image fidelity.

Loss & Training

The overall training process spans two sequential phases: Stage 1 runs for the initial \(60\%\) of total steps (\(S_{\text{warmup}} = 0.6 \cdot S_{\text{total}}\), corresponding to 12,000 steps) optimizing \(\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{base}} + \mathcal{L}_{\text{sem}}\); Stage 2 runs for the subsequent 8,000 steps optimizing \(\mathcal{L}_{\text{stage2}}\) with full-spectrum inputs. Across the entire training duration, the total loss function takes the form:

\[\mathcal{L}_{\text{total}}(s) = \mathcal{L}_{\text{latent}}(s) + \lambda_{\text{pixel}} \cdot \mathcal{L}_{\text{pixel}}(s)\]

Training is conducted on two NVIDIA A100 GPUs with an AdamW optimizer, learning rate \(5 \times 10^{-6}\), and batch size 1 per GPU. A progressive decay schedule is applied to conditioning corruption strength, smoothly transitioning the model from coarse structural disentanglement to fine-grained artistic synthesis.

Key Experimental Results

Main Results

Evaluation is conducted on the newly constructed CSG-Bench benchmark, containing 1,922 content images across portraits, architecture, landscapes, cartoons, objects, food, and animals, evaluated across 6 representative artistic styles (3D Chibi, American Cartoon, Clay, Ghibli, Paper Cutting, Vector). Performance is benchmarked using style-consistency metrics (FID, CSD, CLIP-T) and content-consistency metrics (CLIP-I, DINO, DreamSim).

Table 1: Quantitative results for text-guided style transfer on CSG-Bench.

Method FID ↓ CSD ↑ CLIP-T ↑ CLIP-I ↑ DINO ↑ DreamSim ↑
OmniConsistency (2025) 117.079 0.503 0.266 0.705 0.487 0.712
OmniStyle (2025) 134.415 0.275 0.247 0.737 0.620 0.715
FLUX.1-Kontext-dev (2025) 120.649 0.430 0.251 0.772 0.588 0.741
Nano-banana (2025) 128.162 0.534 0.249 0.792 0.644 0.791
DreamOmni2 (2025) 120.697 0.388 0.252 0.757 0.531 0.704
OmniGen2 (2025) 121.355 0.448 0.260 0.733 0.547 0.733
BAGEL (2025) 132.504 0.525 0.265 0.696 0.425 0.638
Ovis-U1 (2025) 136.033 0.573 0.275 0.702 0.400 0.653
Qwen-Image-Edit (Base) 120.329 0.489 0.251 0.772 0.617 0.761
UniCSG (Ours) 113.940 0.541 0.267 0.797 0.701 0.816

Table 2: Quantitative results for reference-guided style transfer on CSG-Bench.

Method FID ↓ CSD ↑ CLIP-T ↑ CLIP-I ↑ DINO ↑ DreamSim ↑
OmniConsistency (2025) 91.140 0.635 0.269 0.666 0.408 0.679
OmniStyle (2025) 129.775 0.393 0.256 0.732 0.615 0.670
FLUX.1-Kontext-dev (2025) 91.785 0.560 0.255 0.690 0.498 0.685
Nano-banana (2025) 127.868 0.522 0.250 0.745 0.586 0.748
DreamOmni2 (2025) 112.960 0.420 0.242 0.784 0.606 0.748
OmniGen2 (2025) 90.270 0.622 0.271 0.624 0.409 0.624
BAGEL (2025) 72.070 0.511 0.266 0.503 0.212 0.500
Ovis-U1 (2025) 101.215 0.703 0.277 0.552 0.274 0.555
Qwen-Image-Edit (Base) 87.922 0.577 0.253 0.570 0.328 0.565
UniCSG (Ours) 87.320 0.731 0.271 0.760 0.597 0.762

Ablation Study

The ablation experiments systematically isolate the contributions of Stage 1, Stage 2, and the pixel-space reward model under both transfer paradigms.

Table 3: Ablation study on CSG-Bench across text-guided and reference-guided settings.

Configuration Setting FID ↓ CSD ↑ CLIP-T ↑ CLIP-I ↑ DINO ↑ DreamSim ↑
UniCSG (Full Model) Text-guided 113.940 0.541 0.267 0.797 0.701 0.816
w/o Stage 1 (no low-freq disentangle) Text-guided 121.855 0.524 0.258 0.748 0.590 0.763
w/o Stage 2 (no freq-aware detail) Text-guided 124.718 0.516 0.255 0.755 0.630 0.770
w/o Reward (no pixel alignment) Text-guided 123.331 0.521 0.255 0.758 0.622 0.775
UniCSG (Full Model) Reference-guided 87.320 0.731 0.271 0.760 0.597 0.762
w/o Stage 1 (no low-freq disentangle) Reference-guided 90.708 0.645 0.266 0.716 0.417 0.691
w/o Stage 2 (no freq-aware detail) Reference-guided 92.317 0.619 0.262 0.714 0.452 0.701
w/o Reward (no pixel alignment) Reference-guided 92.335 0.661 0.264 0.723 0.451 0.706

Key Findings

  • Stage 1 is the primary line of defense against content leakage: Omitting Stage 1 severely degrades content-consistency metrics, with DINO plummeting from 0.597 to 0.417 and DreamSim dropping from 0.762 to 0.691 in the reference-guided setting. Without low-frequency scaffolding and asymmetric conditioning corruption, strong semantic cues from the reference image directly contaminate the generated content.
  • Stage 2 drives geometric deformation and stroke fidelity: Removing Stage 2 causes a sharp decline in style metrics, with CSD falling from 0.731 to 0.619 in reference-guided transfer. Multi-scale DoG decomposition and LogMSE supervision are essential for rendering category-level artistic stroke textures and structural restyling.
  • Human preference strongly validates perceptual alignment: In a blinded user study with 30 evaluators across 48 cases, UniCSG secured \(38.3\%\) content preference and \(44.6\%\) style preference in reference-guided generation, significantly outperforming competitors such as Nano-banana (\(30.4\% / 24.2\%\)) and FLUX.1-Kontext-dev (\(11.5\% / 7.1\%\)).

Highlights & Insights

  • Asymmetric single-timestep noise amplification: Introducing noise amplification factors \(\gamma_{\text{style}} > \gamma_{\text{content}} > 1\) under a unified diffusion timestep creates a clean information hierarchy without disrupting pretrained noise scheduling or introducing temporal mismatch.
  • Robust high-frequency optimization via LogMSE: Employing LogMSE for high-frequency DoG components effectively dampens gradient explosions from stochastic high-frequency noise while maintaining sharp sensitivity to coherent artistic textures.
  • High inference efficiency: Leveraging Qwen-Image-Lightning distillation, UniCSG completes 1024-resolution generation in approximately 4 seconds for text-guided tasks and 10 seconds for reference-guided tasks, demonstrating high practicality for production deployment.

Limitations & Future Work

  • Performance drop on out-of-distribution artistic styles: When tested on highly specialized unseen styles (e.g., Chinese ink wash, fabric embroidery, origami), reference-guided FID increases to 113.532, indicating that extreme domain shifts still require broader stylistic pretraining data.
  • Computational overhead of pixel-space reward backpropagation: Decoding intermediate latents and backpropagating gradients through multiple external vision encoders substantially increases training GPU memory requirements and wall-clock time. Future work will investigate lightweight latent surrogate reward models.
  • vs. BAGEL / Ovis-U1: While BAGEL attains a lower FID (72.070) in reference-guided transfer, it suffers from catastrophic reference-content leakage, scoring only 0.212 in DINO and 0.500 in DreamSim. UniCSG strikes a vastly superior Pareto frontier between style expression and strict content preservation.
  • vs. UNet-based spatial adapters (e.g., IP-Adapter, ControlNet): Existing adapters enforce rigid pixel-level or shallow spatial constraints that struggle to accommodate category-level geometric restyling (e.g., 3D Chibi or Clay deformations). UniCSG operates natively within DiT representations, enabling both structural shape deformation and high-frequency textural synthesis.

Rating

  • Novelty: ⭐⭐⭐⭐☆ The combination of asymmetric condition corruption and latent DoG multi-scale frequency decomposition offers an elegant, well-principled solution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across CSG-Bench with 6 styles, full ablation tables, user studies, and unseen style stress testing.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous mathematical derivations and clearly articulated empirical motivations.
  • Value: ⭐⭐⭐⭐⭐ Directly resolves the prevalent challenge of reference-content leakage in DiT-based style transfer, with strong practical relevance for production pipelines.