ASTAD: Asymmetric Style Transfer for Auto-driving Synthetic-to-Real Domain Adaptation

Conference: ECCV 2026
arXiv: 2606.29286
Code: https://github.com/Dingyi-Yao/ASTAD
Area: Autonomous Driving / Domain Adaptation / Style Transfer
Keywords: Asymmetric Style Transfer, Synthetic-to-Real Adaptation, Diffusion Models, Semantic Consistency, Prototype Guidance

TL;DR

Addressing the asymmetric constraint in autonomous driving scenarios where "synthetic images have perfect annotations, while real style reference images have no annotations," ASTModel proposes a training-free two-stage framework. It first extracts a coarse semantic prior from unlabeled style images using DINO prototype matching. Then, during the reverse diffusion process, it refines the prior via multi-layer semantic voting, uses robust median threshold filtering and pixel-proportion modulated hybrid AdaIN to achieve class-consistent style injection, generating realistic target-domain style data with a 3.2x speedup while preserving semantic structures.

Background & Motivation

Autonomous driving perception models heavily rely on large-scale, high-density semantic annotations, yet manual labeling in real-world scenarios is extremely expensive (approximately 1.5 hours per Cityscapes image). Synthetic data (such as the GTA dataset), automatically generated via simulators, can quickly provide large-scale datasets with precise pixel-level annotations, presenting an effective approach to solve data scarcity. However, a significant distribution shift exists between synthetic and real-world data due to factors like approximate physical rendering in simulation engines and idealized material textures. This shift causes models trained purely on synthetic data to perform poorly in real-world scenarios. A key approach to bridging this gap is style transfer on synthetic data: preserving its precise layout and annotation while transferring the visual style to that of real-world target scenarios.

Due to their zero-shot generalization capabilities and high-fidelity image generation quality, diffusion models have emerged as the mainstream tool for style transfer in recent years. Existing works (e.g., CACTIF) adopt a "symmetric information assumption," requiring both the content and style images to possess dense semantic segmentation maps. This is used to guide class-wise style transfer and prevent semantic confusion (e.g., ensuring road regions do not acquire vegetation textures). However, this assumption faces a fundamental contradiction in practical deployment: while synthetic content images naturally possess perfect pixel-level annotations, real-world style reference images usually lack any dense annotations. Forcing annotation on style images is highly costly, and failing to do so leads to severe cross-category semantic contamination (e.g., erroneously transferring green vegetation textures onto roads). This information asymmetry represents a key blind spot neglected by prior methods.

The core insight of this paper is: since the style image lacks annotations, is it possible to leverage the existing annotations on synthetic images along with the strong semantic representation capabilities of foundation models to automatically infer semantic priors from unlabeled style images, and then use them to guide style injection? Core Idea: Formulating synthetic-to-real style transfer under asymmetric constraints (annotated synthetic content + unlabeled real style), the authors propose a two-stage training-free framework, ASTModel. Stage I extracts a coarse semantic map from the style image using DINO prototype matching. Stage II refines this prior in diffusion sampling via multi-layer U-Net semantic voting, resolving cross-class leakage and statistical unreliability using robust median threshold filtering and pixel-proportion modulated hybrid AdaIN respectively, thereby ensuring semantically consistent style transfer.

Method

Overall Architecture

ASTModel is a training-free two-stage style transfer framework built on top of a pretrained Latent Diffusion Model (LDM). Stage I is responsible for extracting the semantic structure from unlabeled style images: a frozen DINOv2 encoder is utilized to extract patch-level features from the synthetic content image and the real style image, respectively. Leveraging the known annotations of the content image, class-wise semantic prototype vectors are constructed. Through nearest-neighbor matching in the shared DINO feature space, each location in the style image is assigned the most similar semantic category, yielding a coarse style semantic prior map. Stage II operates during the reverse diffusion sampling process: at each timestep, features are extracted from the high-resolution layers of the U-Net decoder. The prototype matching logic is reused to generate layer-wise instantaneous segmentation predictions, which are then refined through cross-layer consensus voting. Only regions where all layers yield consistent agreement are updated with fine-grained diffusion layer predictions, while other regions maintain the coarse estimate from Stage I. The refined semantic map simultaneously constrains two core injection modules: first, a semantic-constrained adaptive attention filtering, which adaptively filters out sparse high-value correspondences in cross-attention maps using Median/Median Absolute Deviation (Median/MAD) and enforces semantic barriers to only allow intra-class texture injection; second, a Pixel-Proportion Modulated Hybrid AdaIN, which dynamically adjusts the transfer strength of each class based on the ratio of pixel counts of that category in the content and style images. Classes with drastically mismatched pixel proportions undergo less style transfer and retain more of their original content structures. The final output is a stylized image that retains the synthetic image's semantic layout while acquiring target-domain visual features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Annotated Synthetic Image (Ic, Mc)<br/>Unlabeled Style Image (Is)"] --> B["Stage I: Prototype-Guided<br/>Semantic Prior Extraction"]
    B --> B1["DINO Extract Patch Features"]
    B1 --> B2["Synthetic Prototype Construction<br/>Class-wise Feature Averaging"]
    B2 --> B3["Style Image NN Matching<br/>→ Coarse Semantic Prior M_prior"]
    B3 --> C["Stage II: Asymmetric<br/>Style Injection (Diffusion Reverse Process)"]

    C --> D["Multi-layer Semantic Voting<br/>Refines M_prior → M_s^t"]
    D --> E["Semantic-Constrained Adaptive<br/>Attention Filtering"]
    E --> F["Pixel-Proportion Modulated<br/>Hybrid AdaIN"]
    F --> G["Output: Stylized Image<br/>Semantic Layout Preserved + Target Domain Style"]

    D --> D1["U-Net Decoder Multi-layer Feature Extraction"]
    D1 --> D2["Layer-wise Semantic Prediction"]
    D2 --> D3["Cross-layer Consensus Voting<br/>Updated for High Confidence, Maintained for Low Confidence"]
    D3 --> D

    E --> E1["Cross-attention Map → Median/MAD Threshold"]
    E1 --> E2["Semantic Barrier M_s^t<br/>Restricts to Intra-class Texture Injection"]
    E2 --> E3["Soft-gated Fallback Mechanism<br/>Low Confidence Regions Fall Back to Content Features"]
    E3 --> E

    F --> F1["Calculate Class-wise Pixel Ratio \delta_k = (N_c - N_s)/(N_c + N_s)"]
    F1 --> F2["Sigmoid Modulation \beta_k"]
    F2 --> F3["Interpolation: \beta_k \times Content + (1-\beta_k) \times AdaIN Style"]
    F3 --> F

Key Designs

1. Prototype-guided Style Semantic Prior Extraction: Inferring Semantic Structure from Unlabeled Style Images

This is the first core design component to address the asymmetric constraints. Given a completely unlabeled real style image, how can we determine which semantic category each pixel corresponds to? The core idea is to borrow and transfer the annotation knowledge from the synthetic data. First, the frozen DINOv2 is utilized to extract \(L_2\)-normalized patch-level features from both the content and style images. For the content image, leveraging its known semantic annotations, the features are averaged across all pixel locations belonging to each category (e.g., road, building, vegetation) to obtain class prototype vectors. Then, in the shared DINO feature space, for each patch position in the style image, the cosine similarity between it and all class prototypes is calculated. The category corresponding to the highest score is assigned as the coarse semantic label for that position, yielding the semantic prior map.

The advantages of this approach include zero training and zero annotation costs, while DINO's patch features naturally possess semantic alignment capability—similar objects across different images feature highly similar representations. However, the limitation is also evident: DINO features have a low resolution (typically \(16 \times 16\) or \(14 \times 14\) patches), leading to coarse boundaries and a lack of detail in the prior map. Consequently, this coarse prior only serves as a "semantic blueprint" in Stage II, which requires further refinement during the diffusion process.

2. Multi-layer Semantic Voting: Cross-layer Consensus to Refine Semantic Boundaries

The U-Net decoder of a diffusion model contains features at multiple resolution levels. High-resolution layers preserve rich, fine-grained structural information, but their single-layer predictions are unstable during the reverse diffusion process due to stochasticity. Conversely, the Stage I DINO prior is stable but lacks fine boundaries. How does one combine their strengths to compensate for their weaknesses? The proposed mechanism is: at each timestep, style branch features are extracted from a set of high-resolution layers in the decoder. For each layer, the prototype matching logic is reused (using the content features of that layer and synthetic annotations to compute layer-wise prototypes, and then matching style features with nearest neighbors) to obtain an instantaneous semantic prediction for that layer. A consensus mask is then defined: only when all selected layers yield identical classification results at a given position is that location marked as a high-confidence region. The refined semantic map is formulated as: updating high-confidence regions with the fine-grained predictions from diffusion layers, while retaining the coarse DINO prior in low-confidence regions.

The elegance of this design lies in exploiting an inherent property of diffusion models: genuine intrinsic semantic structures of an image consistently manifest across different scales of feature layers, rather than as random noise patterns. Thus, cross-layer consensus naturally acts as a high-precision semantic signal that refines boundaries (utilizing high-resolution details) without introducing stochastic noise from the diffusion process. Experimental results show that this mechanism improves the pseudo-label pixel accuracy from \(0.688\) to \(0.790\) and the mIoU from \(0.281\) to \(0.339\).

3. Semantic-Constrained Adaptive Attention Filtering: Dual Constraints of Robust Thresholds and Semantic Barriers against Cross-class Leakage

Under the asymmetric setting, standard cross-attention mechanisms are highly prone to cross-category texture contamination—the model might incorrectly inject the green texture of vegetation regions in the style image into road regions in the content image. Existing methods (such as CACTIF) filter attention correspondences using a fixed percentile threshold, but assuming a constant signal-to-noise ratio across all images, layers, and timesteps is unreasonable. More fundamentally, the numerical distribution of cross-attention maps exhibits severe heavy-tailed characteristics: valid structural correspondences present themselves as highly sparse, high-value outliers surrounded by a large amount of low-value background noise.

This paper proposes a dual-constraint filtering mechanism. The first layer is the robust statistical threshold: instead of using the traditional Mean \(\pm\) Standard Deviation (which is heavily biased by outliers), the median and Median Absolute Deviation (MAD) are used as the threshold baseline. Since MAD is insensitive to extreme values, it stably captures the background noise level even in the presence of sparse high-value outliers, preventing over-filtering. The dynamic threshold calculated in this manner is more robust than prior fixed-percentile methods—experiments demonstrate that while the Mean/Std threshold retains only \(1.8\%\) of attention entries (leading to under-transfer due to over-filtering), Median/MAD retains \(33.2\%\) of entries, yielding a more complete style transfer. The second layer is the semantic barrier: leveraging the refined semantic map from the previous module, cross-attention connections are only permitted when the category of the content location matches the category of the style location, completely prohibiting cross-class texture injection at the semantic level. The output of the dual-filtered attention is further equipped with a soft-gated fallback mechanism: the ratio of valid correspondences at each location is computed as a retention score \(\alpha\). When \(\alpha\) is high, style textures are heavily injected; when \(\alpha\) is low (no reliable correspondence found in the style image), it falls back to the content features, preventing artifacts from forced correspondences.

4. Pixel-Proportion Modulated Hybrid AdaIN: Dynamically Adjusting Transfer Strength by Category Pixel Abundance

Standard class-wise AdaIN requires precise semantic annotations on both sides. However, under the asymmetric setting, the semantic annotations of style images are inferred and inevitably noisy. A more nuanced issue arises: a certain category might occupy a massive area in the synthetic content image (e.g., roads occupying \(60\%\) of the screen), but appear only in sparse fragments or be completely absent in the style image. Utilizing unreliable statistics to normalize large-area content regions inevitably leads to severe feature distortion.

The solution proposed in this paper is highly intuitive: for each category, the relative difference ratio \(\delta_k = (N_c - N_s) / (N_c + N_s)\) between the content image pixel count \(N_c\) and the style image pixel count \(N_s\) is computed. Then, a Sigmoid function (\(\gamma = 8\)) is used to map \(\delta_k\) to a modulation factor \(\beta_k \in [0,1]\). In the final hybrid AdaIN, the style injection strength is interpolated according to \(\beta_k\): when \(\beta_k\) is close to \(1\) (very few pixels of this class in the style image), content features are primarily preserved; when \(\beta_k\) is close to \(0\) (abundant pixels of this class in the style image), style textures are heavily injected. The core insight of this design is that pixel abundance indirectly reflects the reliability of semantic statistics—the more pixels a class has in the style image, the more trustworthy its mean and variance estimates are, allowing for bold style transfer; otherwise, a conservative approach of preserving original structures is preferred.

A Complete Example: Style Transfer in a Vegetation-Dominated Scenario

Consider a GTA synthetic road scene (content image \(I_c\) with complete semantic annotations \(M_c\)) and a real-world street photo dominated by dense vegetation (style image \(I_s\), unlabeled). In Stage I, DINOv2 extracts patch features for both images, where GTA annotations specify which patches belong to categories like road, building, vegetation, sky, and vehicles, thereby constructing semantic prototypes. For each patch of the style image, matching with the most similar category yields a coarse semantic map—it roughly distinguishes the sky at the top, vegetation on both sides, and the road at the bottom, but with coarse boundaries. In Stage II, the reverse diffusion runs step-by-step across 50 timesteps: at step \(t\), style features are extracted from the high-resolution layers of the U-Net decoder (e.g., layers 8, 9, 10) to perform NN matching independently. The coarse prior is updated with fine predictions only when all three layers agree that a location is "road", otherwise the coarse prior is maintained. The refined semantic map then drives attention filtering: only cross-attention links such as "road \(\rightarrow\) road" and "vegetation \(\rightarrow\) vegetation" are allowed, with low-similarity links filtered out using the Median/MAD threshold (removing substantial low-value noise). Finally, hybrid AdaIN calculates the pixel ratios for each class in the scene: vegetation accounts for about \(30\%\) of the content image and \(50\%\) of the style image, representing a small discrepancy (\(\beta_{\text{vegetation}} \approx 0.3\)), hence the green texture of vegetation is fully transferred; conversely, cars or traffic signs in the content image barely exist in the style image (\(\beta_{\text{small\_targets}} \approx 1\)), meaning these regions undergo almost no style transfer and remain intact. The final generated result preserves the full layout and sharp boundaries of GTA while exhibiting real-world material textures in the road, vegetation, and building regions.

Key Experimental Results

Main Results

Dataset	Metric	Source Only	Cross-Image Attn.	CACTIF	ASTModel (Ours)
GTA→Cityscapes (SegFormer)	Pixel Acc ↑	0.844	0.653	0.782	0.847
GTA→Cityscapes (SegFormer)	mIoU ↑	0.275	0.224	0.289	0.309
Structural Fidelity	LPIPS ↓	—	0.5205	0.4184	0.3588
Inference Speed	Single-Image Time ↓	—	18s	80s	25s
Inference Speed	5000-Image Total Time ↓	—	28h	114h	37h

The main experiments evaluate three dimensions: downstream perception utility (via SegFormer segmentation), structural fidelity (LPIPS), and computational efficiency. ASTModel outperforms the Source Only baseline on downstream segmentation metrics (Pixel Acc \(+0.3\), mIoU \(+3.4\)), indicating that the stylized data indeed facilitates domain adaptation; it also yields significant improvements over CACTIF, alongside a substantially reduced LPIPS (\(0.3588\) vs. \(0.4184\)), indicating stronger semantic consistency and fewer structural distortions. In terms of speed, CACTIF is bottlenecked by dense pixel-level feature similarity computations (\(80\text{s}/\text{image}\)), whereas ASTModel's robust statistical thresholding bypasses these intensive computations, achieving a 3.2x speedup.

Ablation Study

Configuration	LPIPS ↓	Description
Full Model	0.3588	All modules active
w/o Attention Filtering	0.5114	No filtering \(\rightarrow\) cross-class contamination, green road artifacts
w/o Hybrid AdaIN	0.3737	Global AdaIN \(\rightarrow\) color shift, decreased fidelity
w/o Semantic Voting	0.3641	Coarse prior cannot be refined, blurry small objects/boundaries

Key Findings

• Attention filtering contributes the most: Removing attention filtering causes LPIPS to surge from \(0.3588\) to \(0.5114\), demonstrating that cross-class semantic leakage is the most critical issue, and semantic-constrained attention filtering is the key to resolving it.
• Robustness of Median/MAD: After removing the top-20 highest attention scores, the Mean/Std remained almost unchanged (Mean from \(41.8 \rightarrow 41.8\), Std from \(173.3 \rightarrow 171.9\), as outliers still dominate the statistics), whereas the Median and MAD were unaffected. The Mean/Std threshold retained only \(1.8\%\) of entries (causing under-transfer), while Median/MAD retained \(33.2\%\) of entries for more complete transfer.
• Validation of pseudo-label refinement: Stage II semantic voting improves the pseudo-label pixel accuracy from \(0.688\) to \(0.790\) and mIoU from \(0.281\) to \(0.339\). The improvement is positively correlated with the semantic compatibility between content and style—the more similar the scenes are, the stronger the cross-layer consensus and the better the refinement.

Highlights & Insights

• Formal definition of asymmetric constraints is the core contribution: Prior methods performed style transfer under the "symmetric information assumption." This work is the first to explicitly define the practical constraint of "annotation asymmetry" in synthetic-to-real adaptation, establishing a more realistic problem formulation for subsequent methods. This task definition itself may be a more substantial contribution than the specific method details.
• Prior refinement via cross-layer consensus voting is highly ingenious: Leveraging the natural properties of the multi-scale features in the diffusion U-Net decoder (rich detailed features in lower layers, stable semantic features in higher layers) for refinement via cross-layer consistency allows a "zero-cost" refinement approach without extra training or annotations.
• Replacing Mean-Std with MAD for attention thresholding: Under heavy-tailed distributions, Median and MAD are robust estimators unaffected by outliers. This insight is not only transferable to other vision tasks requiring adaptive thresholds (e.g., image matching, feature selection, correspondence filtering) but is also intuitive.
• The design of Pixel-Proportion Modulated AdaIN is highly practical: Rather than analyzing the scale of feature distribution differences, it directly queries "whether the pixel count is statistically reliable." This criterion is conceptually sound and extremely simple to implement (using a single Sigmoid).
• Practical value of the training-free framework: By bypassing the need to train specialized style transfer models for each target domain, inference can be performed directly given a reference image. This holds significant value for practical deployment across diverse target domains in autonomous driving.

Limitations & Future Work

• Dependency on a single style reference image: The paper acknowledges this as the primary limitation. A single image may contain very few pixels for certain classes or lack them entirely, leading to insufficient style transfer or feature distortions in those areas. The observation that refinement is strong for rear vehicle views but weaker for front views is highly valuable—extending to multiple reference images in the future could mitigate this issue.
• Sensitivity to scene semantic compatibility: The text notes that the pseudo-label refinement quality is positively correlated with the semantic compatibility of the content and style images. This implies that when the scene discrepancy is too large (e.g., pairing a GTA urban street scene with a desert style image), the refinement quality might degrade. The robustness boundary under extreme scenarios is not thoroughly discussed.
• Lower bound of DINO prior quality: Stage I priors depend entirely on DINO's semantic alignment capability. In scenarios where DINO struggles to distinguish classes (e.g., truck vs. bus), the prior itself introduces confusion that subsequent refinement cannot fully correct. This could pose challenges for fine-grained vehicle classification (car/SUV/truck) in autonomous driving.
• Downstream task evaluation can be more comprehensive: Currently, validation is limited to semantic segmentation (SegFormer). Further validation on object detection (e.g., YOLO) and instance segmentation would make the paper's conclusions more convincing.
• The advantages of training-free frameworks are also their limitations: Unable to leverage target-domain data for fine-tuning to further bridge the domain gap, these methods may face a quality ceiling under extreme domain shifts. A hybrid scheme combining "training-free inference + partial adaptation fine-tuning" could be explored in future work.

Related Work & Insights

• vs CACTIF: CACTIF assumes dense annotations exist on both sides (symmetric assumption) and uses a fixed percentile for attention filtering alongside standard class-wise AdaIN. ASTModel removes the annotation requirement for the style image and enhances the robustness of filtering and normalization via Median/MAD and pixel-proportion modulated AdaIN, respectively. ASTModel significantly outperforms CACTIF on downstream segmentation and LPIPS, and is 3.2x faster.
• vs Cross-Image Attention: Simply exchanging self-attention key/values achieves style transfer but lacks semantic guidance, leading to severe semantic confusion (\(mIoU\) of only \(0.224\)) under asymmetric settings. This indicates that for semantically dense scenarios like autonomous driving, unguided attention exchange is insufficient to guarantee semantic consistency.
• vs DGInStyle / SimGen / Weather-Diff: These methods fine-tune diffusion models to achieve domain-specific style transfer. While they deliver high quality, they lack flexibility. ASTModel's training-free paradigm possesses unique advantages in flexibility and efficiency, making it well-suited for scenarios requiring rapid adaptation to diverse target domains.

Rating

• Novelty: ⭐⭐⭐⭐½ Formulating the "asymmetric constraint in synthetic-to-real adaptation" (a previously overlooked problem) is highly original, with solid task definition and method design. The dual-filtering concept utilizing robust statistics and semantic barriers is simple and effective.
• Experimental Thoroughness: ⭐⭐⭐⭐ Evaluates downstream segmentation, structural fidelity, and inference speed, with comprehensive ablation studies and targeted analysis on pseudo-label refinement and threshold selection. However, the evaluation scope is somewhat narrow, relying on 5,000 GTA images and a single semantic segmentation model.
• Writing Quality: ⭐⭐⭐⭐½ Clear motivation, coherent method formulation, and intuitive visualizations (attention distribution, \(\alpha\) scores, and pseudo-label refinement comparison) supporting key design choices. Balanced equation usage and fluent narration.
• Value: ⭐⭐⭐⭐½ The task formulation is universally relevant—almost all synthetic-to-real adaptation tasks suffer from annotation asymmetry. The training-free framework possesses high practical deployment value, highlighted by a significant 3.2x speedup. Even with certain limitations, the task definition itself warrants substantial attention from subsequent research.

Related Papers

• [ECCV 2026] HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training
• [ECCV 2026] Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout
• [ECCV 2026] UniTeD: Unified Temporal Diffusion for Joint Perception and Planning in Autonomous Driving
• [ECCV 2026] Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints
• [ECCV 2026] FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction