ASTAD: Asymmetric Style Transfer for Synthetic-to-Real Domain Adaptation in Autonomous Driving¶

Conference: ECCV 2026
arXiv: 2606.29286
Code: https://github.com/Dingyi-Yao/ASTAD
Area: Autonomous Driving / Domain Adaptation / Style Transfer
Keywords: Asymmetric Style Transfer, Synthetic-to-Real Adaptation, Diffusion Models, Semantic Consistency, Prototype Guidance

TL;DR¶

Addressing the asymmetric constraint in autonomous driving scenarios where "synthetic images possess perfect annotations, whereas real-world style reference images are unannotated," ASTModel proposes a two-stage training-free framework. It first extracts coarse semantic priors from unlabeled style images via DINO prototype matching. During the reverse diffusion process, these priors are refined through multi-layer semantic voting, followed by class-consistent style injection enabled by robust median-threshold filtering and pixel-proportion modulated hybrid AdaIN. This generates realistic target-domain style data while preserving semantic structures, achieving a 3.2x speedup.

Background & Motivation¶

Autonomous driving perception models heavily rely on large-scale, high-density semantic annotation data, yet the manual annotation of real-world scenes is extremely costly (approximately 1.5 hours per Cityscapes image). Synthetic data (such as the GTA dataset) generated automatically by simulators offers rapid access to large-scale datasets with precise pixel-level annotations, presenting an effective path to address data scarcity. However, a significant distribution shift exists between synthetic data and the real world. Due to the approximated physical rendering of simulation engines and idealized material textures, models trained solely on synthetic data suffer severe performance degradation in real scenarios. A key approach to bridging this gap is style transfer on synthetic data: preserving its precise layout and annotations while transferring the visual style to that of real-world target domains.

Diffusion models have become the mainstream tool for style transfer in recent years due to their zero-shot generalization capabilities and high-fidelity image generation quality. Existing works (such as CACTIF) adopt a "symmetric information assumption," requiring both content and style images to possess dense semantic segmentation maps. This guides style transfer class-by-class to prevent semantic confusion (e.g., ensuring road areas are not stylized with vegetation textures). However, this assumption encounters a fundamental contradiction in practical deployment: synthetic content images naturally possess perfect pixel-level annotations, but real-world style reference images typically lack any dense annotations. Forcing annotation on style images is highly expensive, while failing to do so leads to severe cross-category semantic leakage (such as incorrectly transferring green vegetation textures to roads). This information asymmetry is a critical blind spot overlooked by previous methods.

The core insight of this study is: since style images lack annotations, can the existing annotations on synthetic images and the strong semantic representation capabilities of foundation models be leveraged to automatically infer semantic priors from unlabeled style images, and then use them to guide style injection? Core Idea: Operating under the asymmetric constraint of synthetic-to-real style transfer (annotated synthetic content + unlabeled real-world style), this paper proposes ASTModel, a two-stage training-free framework. Stage I extracts a coarse semantic map from the style image using DINO prototype matching, and Stage II refines this prior via multi-layer U-Net semantic voting during diffusion sampling. It resolves cross-class leakage and statistical unreliability using robust median-threshold filtering and pixel-proportion modulated hybrid AdaIN, respectively, ensuring semantically consistent style transfer.

Method¶

Overall Architecture¶

ASTModel is a training-free, two-stage style transfer framework built upon a pretrained Latent Diffusion Model (LDM). Stage I is responsible for extracting semantic structure from unlabeled style images: a frozen DINOv2 encoder is used to extract patch-level features from the synthetic content image and the real style image, respectively. Utilizing the known annotations of the content image, class-wise semantic prototype vectors are constructed. In the DINO shared feature space, nearest-neighbor matching is performed to assign each position of the style image to its most similar semantic category, yielding a coarse style semantic prior map. Stage II operates during the reverse diffusion sampling process: at each timestep, features are extracted from the high-resolution layers of the U-Net decoder, and the prototype matching logic is reused to generate layer-wise instantaneous segmentation predictions. These predictions then refine the prior through cross-layer consensus voting—only regions where all layers yield a unanimous consensus are updated with the fine-grained diffusion-layer predictions, while the remaining regions maintain the coarse Stage I estimation. The refined semantic map simultaneously constrains two core injection modules: first, semantically-constrained adaptive attention filtering, which adaptively filters sparse high-value correspondences in cross-attention maps using median and median absolute deviation (Median/MAD), and enforces a semantic barrier to allow style injection only between identical classes; second, Pixel-Proportion Modulated Hybrid AdaIN, which dynamically modulates the style transfer intensity of each class according to the pixel count ratio of that class between the content and style images. Classes with highly disparate pixel proportions undergo less style transfer, preserving their original content structures. The final output is a stylized image that retains the synthetic semantic layout while exhibiting the visual characteristics of the target domain.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Annotated Synthetic Image (Ic, Mc)<br/>Unlabeled Style Image (Is)"] --> B["Stage I: Prototype-Guided<br/>Semantic Prior Extraction"]
    B --> B1["DINO Feature Extraction"]
    B1 --> B2["Synthetic Prototype Construction<br/>(Class-wise Feature Averaging)"]
    B2 --> B3["Style Image NN Matching<br/>→ Coarse Semantic Prior M_prior"]
    B3 --> C["Stage II: Asymmetric<br/>Style Injection (Reverse Diffusion)"]

    C --> D["Multi-Layer Semantic Voting<br/>Refining M_prior → M_s^t"]
    D --> E["Semantically-Constrained Adaptive<br/>Attention Filtering"]
    E --> F["Pixel-Proportion Modulated<br/>Hybrid AdaIN"]
    F --> G["Output: Stylized Image<br/>(Preserved Layout + Target Domain Style)"]

    D --> D1["U-Net Decoder Feature Extraction"]
    D1 --> D2["Layer-wise Semantic Prediction"]
    D2 --> D3["Cross-Layer Consensus Voting<br/>(High-confidence updates, low-confidence remains)"]
    D3 --> D

    E --> E1["Cross-Attention Map → Median/MAD Thresholding"]
    E1 --> E2["Semantic Barrier M_s^t<br/>(Restricts context to same-class texture)"]
    E2 --> E3["Soft Gating Rollback Mechanism<br/>(Fallback to content feature for low confidence)"]
    E3 --> E

    F --> F1["Calculate Class-wise Pixel Ratio δ_k = (N_c - N_s)/(N_c + N_s)"]
    F1 --> F2["Sigmoid Modulation β_k"]
    F2 --> F3["Interpolation: β_k × Content + (1-β_k) × AdaIN Style"]
    F3 --> F

Key Designs¶

1. Prototype-Guided Semantic Prior Extraction: Inferring Semantic Structure from Unlabeled Style Images

This is the first core design component to tackle the asymmetric constraint. Given a completely unlabeled real-world style image, how can the semantic category of each pixel be identified? The core mechanism is to borrow the annotation knowledge from the synthetic data. First, a frozen DINOv2 is used to extract L2-normalized patch-level features from the content and style images, respectively. For the content image, leveraging its known semantic annotations, the features of all pixel positions belonging to each category (e.g., road, building, vegetation) are averaged to obtain class prototype vectors. In the DINO shared feature space, for each patch location of the style image, the cosine similarity with all class prototypes is calculated. The category with the highest similarity score is assigned as the coarse semantic label for that position, producing the semantic prior map.

The advantage of this method is its zero-training and zero-annotation costs, alongside the inherent semantic alignment capability of DINO patch features, as identical objects across different images exhibit highly similar features. However, the limitation is also evident: DINO features have a low resolution (typically \(16 \times 16\) or \(14 \times 14\) patches), leading to a coarse prior map with rough boundaries and lacking details. Hence, this coarse prior only serves as a "semantic blueprint" for Stage II and must be further refined during the diffusion process.

2. Multi-Layer Semantic Voting: Cross-Layer Consensus for Boundary Refinement

The diffusion model's U-Net decoder contains feature maps at multiple resolution levels: high-resolution layers preserve rich, fine-grained structural information, but individual layer predictions during the reverse diffusion are unstable due to stochasticity. Conversely, the Stage I DINO prior is stable but lacks precise boundaries. To combine their strengths and compensate for their weaknesses, a new mechanism is proposed: at each timestep, style branch features are extracted from a set of high-resolution decoder layers. For each layer, the prototype matching logic is reused (calculating layer-wise prototypes based on content features and synthetic annotations of that layer, followed by NN matching on style features) to obtain instantaneous semantic predictions. A consensus mask is then defined—a position is flagged as a high-confidence zone only when all selected layers yield identical classification results. The refined semantic map is formulated such that high-confidence zones adopt the fine-grained predictions of the diffusion layers, while low-confidence zones maintain the coarse DINO prior.

The elegance of this design lies in leveraging an intrinsic property of diffusion models: authentic, inherent semantic structures of an image manifest consistently across feature layers of different scales, unlike random noise patterns. Consequently, cross-layer consensus naturally serves as a high-precision semantic signal, refining boundaries using high-resolution details without introducing stochastic noise. Experiments indicate that this mechanism improves pseudo-label pixel accuracy from 0.688 to 0.790, and mIoU from 0.281 to 0.339.

3. Semantically-Constrained Adaptive Attention Filtering: Dual Constraints of Robust Thresholding and Semantic Barriers Against Cross-Class Leakage

Under the asymmetric setting, standard cross-attention mechanisms are highly prone to cross-category texture contamination—the model might incorrectly inject green vegetation textures from the style image into road regions of the content image. Existing methods (such as CACTIF) filter attention correspondences using fixed percentile thresholds, which unrealistically assumes a constant signal-to-noise ratio across all images, layers, and timesteps. More fundamentally, the distribution of values in cross-attention maps exhibits a heavy-tailed property: valid structural correspondences appear as extremely sparse, high-value outliers surrounded by a large volume of low-value background noise.

This paper proposes a dual-constraint filtering mechanism. The first is a robust statistical threshold: instead of using the traditional Mean \(\pm\) Standard Deviation (which is heavily biased by outliers), the median and median absolute deviation (MAD) are adopted as the threshold baseline. Since MAD is robust to extreme values, it stably captures the background noise level even in the presence of sparse high-value outliers, preventing over-filtering. The resulting dynamic threshold is far more robust than prior fixed-percentile/standard deviation approaches—experiments demonstrate that Mean/Std thresholding retains only 1.8% of the attention entries (under-transfer due to over-filtering), whereas Median/MAD retains 33.2%, achieving a more complete transfer. The second constraint is a semantic barrier: leveraging the refined semantic map from the previous module, attention connections are permitted only when the category of the content position matches that of the style position, completely blocking cross-class texture leakage at the semantic layer. The filtered attention output is further equipped with a soft-gating rollback mechanism: the proportion of valid correspondences at each location is computed as a retention score \(\alpha\). When \(\alpha\) is high, style textures are heavily injected; when \(\alpha\) is low (indicating a lack of reliable correspondences in the style image), features fall back to the original content to avoid artifacts.

4. Pixel-Proportion Modulated Hybrid AdaIN: Dynamically Tuning Transfer Intensity by Class Pixel Abundance

Standard class-wise AdaIN requires precise semantic annotations on both sides. However, under the asymmetric setup, the style annotations are inferred and inherently noisy. A more nuanced issue arises when a class covers a huge area in the synthetic content image (e.g., roads occupying 60% of the frame) but is scarcely present or entirely absent in the style image—normalizing a massive content region using unreliable style statistics inevitably causes severe feature distortion.

The proposed solution is straightforward: for each category, the relative discrepancy ratio of the pixel count is calculated as \(\delta_k = (N_c - N_s) / (N_c + N_s)\), which is then mapped to a modulation factor \(\beta_k \in [0, 1]\) via a Sigmoid function (\(\gamma=8\)). In the final hybrid AdaIN step, the style injection intensity is interpolated based on \(\beta_k\): when \(\beta_k\) is close to 1 (indicating few pixels of this class in the style image), content features are primarily preserved; when \(\beta_k\) approaches 0 (the class is abundant in the style image), styled features are fully integrated. The key insight of this design is that pixel abundance indirectly reflects the reliability of semantic statistics—the more pixels a class has in the style image, the more trustworthy its mean/variance estimates are, allowing for confident style transfer; otherwise, a conservative strategy preserving the original structure is preferred.

A Complete Example: Style Transfer in a Vegetation-Dominated Scene¶

Consider a synthetic GTA road scene (content image \(I_c\) with complete semantic annotations \(M_c\)) and a real-world street photo with dense vegetation (style image \(I_s\), unlabeled). In Stage I, DINOv2 extracts patch features for both images, and the GTA annotations indicate which patches belong to roads, buildings, vegetation, sky, vehicles, etc., thereby constructing the semantic prototypes. Each patch of the style image is matched to its most similar category, resulting in a coarse semantic map—roughly identifying sky at the top, vegetation on both sides, and roads at the bottom, albeit with coarse boundaries. Stage II runs timestep-by-timestep over 50 steps of diffusion sampling: at step \(t\), style features are extracted from high-resolution layers of the U-Net decoder (e.g., layers 8, 9, 10) and subjected to NN matching. Only when all three layers agree that a location is "road" is the fine-grained prediction used to update the prior; otherwise, the coarse prior is maintained. The refined semantics then drive attention filtering: cross-attention connections are permitted only for identical classes (e.g., "road \(\to\) road", "vegetation \(\to\) vegetation"), and any low-similarity connections are discarded using the Median/MAD threshold (filtering out substantial low-value noise). Finally, hybrid AdaIN computes pixel proportions for each class in the scene: vegetation accounts for ~30% in the content image and ~50% in the style image, showing minimal discrepancy (\(\beta_{\text{vegetation}} \approx 0.3\)), which allows for a robust transfer of the vegetation texture. Conversely, small targets like cars and traffic signs in the content image are practically non-existent in the style image (\(\beta_{\text{small\_targets}} \approx 1\)), meaning these regions undergo almost no transfer and preserve their original textures. The final generated image retains the complete layout and sharp boundaries of the GTA image while conveying the realistic material textures of the real world in the road, vegetation, and building regions.

Key Experimental Results¶

Main Results¶

Dataset	Metric	Source Only	Cross-Image Attn.	CACTIF	ASTModel (Ours)
GTA→Cityscapes (SegFormer)	Pixel Acc ↑	0.844	0.653	0.782	0.847
GTA→Cityscapes (SegFormer)	mIoU ↑	0.275	0.224	0.289	0.309
Structural Fidelity	LPIPS ↓	—	0.5205	0.4184	0.3588
Inference Speed	Per-image Time ↓	—	18s	80s	25s
Inference Speed	Total Time (5000 images) ↓	—	28h	114h	37h

The main experiments evaluate three dimensions: downstream perception utility (via SegFormer segmentation), structural fidelity (LPIPS), and computational efficiency. ASTModel outperforms the Source Only baseline on downstream segmentation metrics (Pixel Acc +0.3%, mIoU +3.4%), demonstrating that the stylized data effectively aids domain adaptation. Compared to CACTIF, ASTModel shows substantial gains, and its LPIPS is significantly lower (0.3588 vs. 0.4184), indicating stronger semantic consistency and fewer structural distortions. Regarding speed, while CACTIF encounters a performance bottleneck due to pixel-wise feature similarity calculations (80s per image), ASTModel's robust statistical thresholding bypasses such dense computations, achieving a 3.2x acceleration.

Ablation Study¶

Configuration	LPIPS ↓	Description
Full Model	0.3588	All modules operating simultaneously
w/o Attention Filtering	0.5114	No filtering \(\to\) cross-class leakage, resulting in green artifacts on roads
w/o Hybrid AdaIN	0.3737	Global AdaIN \(\to\) color shift and degraded fidelity
w/o Semantic Voting	0.3641	Coarse prior remains unrefined, blurring small objects and boundaries

Key Findings¶

Attention Filtering Contributes the Most: Removing the attention filtering component causes LPIPS to surge from 0.3588 to 0.5114, indicating that cross-class semantic leakage is the most critical issue and that semantically-constrained attention filtering is the key to resolving it.
Robustness of Median/MAD: After removing the top-20 highest attention scores, the mean remains at 41.8 \(\to\) 41.8 and the standard deviation changes from 173.3 \(\to\) 171.9 (practically unchanged as outliers still dominate the statistics), whereas the median and MAD are unaffected. Traditional Mean/Std thresholding only retains 1.8% of entries, leading to under-stylization, whereas Median/MAD retains 33.2%, achieving a more complete style transfer.
Pseudo-label Refinement Validation: Stage II semantic voting improves pseudo-label pixel accuracy from 0.688 to 0.790, and mIoU from 0.281 to 0.339. The degree of improvement is positively correlated with the semantic compatibility between content and style—the more similar the scenes, the stronger the cross-layer consensus, and the better the refinement.

Highlights & Insights¶

Formalizing the asymmetric constraint is a core contribution: Prior methods operated under "symmetric information assumptions" for style transfer. This paper is the first to explicitly define the "annotation asymmetry" constraint in synthetic-to-real adaptation, establishing a more realistic problem framework for future works. This task formulation itself is arguably a greater contribution than the specific method.
Elegant prior refinement using cross-layer consensus voting: By utilizing the natural properties of the diffusion U-Net decoder at multiple feature scales (fine details in bottom layers, stable semantics in top layers), refinement is executed via cross-layer consistency without requiring training or additional annotations, serving as a "zero-cost" refining strategy.
Replacing mean-std with MAD for attention thresholding: In heavy-tailed distributions, the median and MAD are robust statistics unaffected by outliers. This insight is highly intuitive and can be generalized to other vision tasks requiring adaptive thresholding (such as image matching, feature selection, and correspondence filtering).
Highly operational pixel-proportion modulated AdaIN: Assessing whether a class's pixel abundance is "statistically reliable" rather than analyzing feature distribution discrepancy is a conceptually solid and straightforward criterion to implement (utilizing a simple Sigmoid).
Practical value of a training-free framework: It eliminates the need to train specialized style transfer models for each target domain, allowing immediate inference simply by swapping the reference image. This is highly suitable for practical deployment across diverse target domains in autonomous driving.

Limitations & Future Work¶

Reliance on a single style reference image: The paper acknowledges this as the most prominent limitation. Certain classes in a single image may have few pixels or be completely absent, leading to insufficient style transfer or feature distortion. Addressing this by extending the framework to leverage multiple reference images is a promising direction.
Sensitivity to scene semantic compatibility: As noted, the pseudo-label refinement effectiveness is positively correlated with the semantic compatibility of the content and style images. Consequently, the refinement might degrade when scene differences are extreme (e.g., matching a GTA city street with a desert image). The robustness boundaries under such extreme scenarios are not extensively discussed.
Lower bound of DINO prior quality: Stage I priors entirely depend on DINO's semantic alignment capability. When encountering categories that are difficult for DINO to distinguish (such as trucks vs. buses), the initial prior itself introduces confusion that Stage II refinement cannot fully rectify. This could be a bottleneck for fine-grained vehicle classification (e.g., cars vs. SUVs vs. trucks) in autonomous driving.
Desirability of broader downstream task evaluations: Currently, validation is limited to semantic segmentation (SegFormer). Verifying performance on other tasks such as object detection (e.g., YOLO) and instance segmentation would make the conclusions more persuasive.
Trade-off of the training-free paradigm: The framework cannot exploit target-domain target data for fine-tuning to further bridge the domain gap, potentially imposing a quality ceiling in extreme domain shift scenarios. A hybrid pipeline combining training-free inference with localized adaptation fine-tuning could be investigated.

vs. CACTIF: CACTIF assumes dense annotations for both content and style images (symmetric assumption), utilizes fixed percentile thresholds for filtering attention, and implements standard class-wise AdaIN. ASTModel removes the requirement for style image annotations and enhances filtering and normalization robustness through Median/MAD and pixel-proportion modulated AdaIN, respectively. ASTModel significantly outperforms CACTIF in downstream segmentation metrics and LPIPS, while running 3.2x faster.
vs. Cross-Image Attention: Simple self-attention key/value swapping across images lacks semantic guidance and leads to severe semantic confusion under the asymmetric setting (mIoU of only 0.224). This indicates that unguided attention swapping is insufficient to maintain semantic consistency in semantically dense scenarios like autonomous driving.
vs. DGInStyle / SimGen / Weather-Diff: These methods fine-tune diffusion models to achieve domain-specific style transfer, yielding high quality but low flexibility. ASTModel's training-free paradigm holds unique advantages in flexibility and efficiency, making it well-suited for scenarios requiring rapid adaptation to diverse target domains.

Rating¶

Novelty: ⭐⭐⭐⭐½ First to formalize the overlooked "asymmetric constraint in synthetic-to-real adaptation." Both the task definition and method design show high originality. The dual constraints of robust statistics and semantic barriers are elegant and effective.
Experimental Thoroughness: ⭐⭐⭐⭐ Covers downstream segmentation, structural fidelity, and inference speed. The ablation study is comprehensive, with specialized analyses on pseudo-label refinement and threshold choices. However, evaluating on only 5,000 GTA images and one semantic segmentation model is somewhat narrow.
Writing Quality: ⭐⭐⭐⭐½ The motivation is clear, the methodological derivation is logical, and the critical design choices are backed by intuitive visualizations (e.g., attention distributions, \(\alpha\) scores, pseudo-label refinement comparison maps). The amount of mathematics is balanced and the narrative is fluent.
Value: ⭐⭐⭐⭐½ The task formulation is highly generalizable, as nearly all synthetic-to-real adaptation settings implicitly face annotation asymmetry. High practical utility of the training-free framework is demonstrated, alongside a significant 3.2x speedup. Even with the method's current limitations, the defined paradigm deserves field-wide attention.