Rethinking Cross-Spectral Image Generation via Shared-Specific Representation

Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Cross-Spectral Image Generation, Visible-to-Infrared, Shared-Specific Representation, Feature Decoupling and Reconstruction, Semantic Alignment

TL;DR

Addressing semantic misalignment and distorted thermal properties in visible-to-infrared cross-spectral translation, this paper proposes SHASP, which reframes cross-modal synthesis as a latent decoupling and reconstruction process of shared structural and modality-specific spectral features, delivering state-of-the-art structural consistency and thermal fidelity.

Background & Motivation

Visible (VIS) and infrared (IR) sensing constitute two indispensable, complementary data modalities in computer vision. Visible sensors capture sharp surface reflectance and rich geometric details, whereas infrared sensors record thermal radiation patterns and temperature distributions. Synergizing both modalities is pivotal for safety-critical downstream tasks such as all-weather autonomous driving, night-time security surveillance, multi-spectral object detection, and person re-identification. However, in practical deployment scenarios, collecting large-scale, well-calibrated infrared training data is prohibitively expensive and logistically challenging. Consequently, synthesizing realistic, physically plausible infrared data from readily accessible visible imagery has emerged as an essential data augmentation pathway to alleviate the infrared data scarcity bottleneck.

Despite significant progress in generative adversarial networks (GANs) and diffusion models, existing cross-spectral translation approaches face severe representation bottlenecks. Most current methods formulate cross-modal translation as an end-to-end pixel-level nonlinear style mapping. This global, unconstrained mapping fails to explicitly decouple shared geometric content from unique physical attributes, frequently resulting in geometric distortions, blurred object boundaries, severe semantic misalignment, and missing thermal characteristics. Although patch-wise contrastive learning paradigms attempt to impose local semantic constraints, the vast domain discrepancy between visible reflectance and thermal radiation prevents low-level pixel constraints from ensuring semantic structure consistency, leaving synthesized infrared images over-smoothed or plagued by synthetic artifacts.

From the perspectives of physics and visual sensing theory, although visible and infrared images diverge fundamentally in low-level radiation mechanisms, they demonstrate pronounced statistical consistency and geometric invariance in deep semantic structures—such as spatial scene layouts, object contour boundaries, and part arrangements. In fact, deep feature discrepancies between the two modalities are substantially smaller than the energy of their shared structural representations. This observation motivates reformulating the synthesis task: Core idea: reframe cross-spectral image generation as an explicit representation learning problem of disentangling and reconstructing shared structural content and modality-specific spectral attributes, leveraging dual-branch encoding and adversarial semantic refinement in the latent space for high-fidelity infrared synthesis.

Method

Overall Architecture

The proposed SHASP (SHAred-SPecific generative network) architecture consists of a bidirectional translation loop comprising forward generator \(G\) (VIS \(\to\) IR), reverse generator \(P\) (IR \(\to\) VIS), target domain discriminators \(D_X\) and \(D_Y\), and a semantic structure discriminator \(D_{str}\). Within each generator, the pipeline is divided into an Encoding-Decoupling Module and a Decoding-Reconstruction Module. In the encoding phase, input images are routed into a dual-branch structure content encoder and modality-dedicated spectral feature encoders to disentangle domain-invariant spatial semantics from exclusive spectral attributes. In the decoding-reconstruction phase, the decoupled feature vectors are concatenated along the channel dimension, fused through multi-level residual blocks, and upsampled via transposed convolutions to reconstruct target images. Meanwhile, a semantic structure discriminator enforces domain-adversarial constraints on the shared structural representations, eliminating residual modal bias and guaranteeing rigorous cross-modal semantic alignment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Images<br/>Visible VIS / Infrared IR"] --> B["Structure Content Encoder<br/>Multi-scale downsampling + shared residual layers"]
    A --> C["Reverse-Pyramid Spectral Encoder<br/>Multi-scale conv + adaptive global pooling"]
    B -->|"Shared structural features F_shared"| D["Decoding-Reconstruction Module<br/>Channel concatenation + residual blocks & deconv"]
    C -->|"Modality-specific features F_spec"| D
    B -.->|"Domain-adversarial constraint"| E["Semantic Structure Refinement<br/>Structure discriminator D_str supervises invariance"]
    D --> F["Synthesized Cross-Spectral Images<br/>Equivalent IR / VIS images"]
    F --> G["Bidirectional Cycle & Adversarial Losses<br/>Cycle consistency + domain discriminators DX/DY"]

Key Designs

1. Structure Content Encoder: Weight-Shared Domain-Invariant Semantic Decoupling

Standard encoders struggle to disentangle multi-modal heterogeneity, inevitably introducing geometric distortions into synthesized outputs. To extract modality-invariant geometric structures, object contours, and spatial layout representations, the structure content encoder \(En_{str}\) employs a dual-branch encoding architecture. Each branch initiates feature extraction using large \(7\times 7\) convolution kernels with nonlinear activations to preserve fine spatial details with an expansive receptive field, followed by cascaded \(3\times 3\) convolutions for spatial downsampling and channel expansion, producing high-level semantic representations \(F_{VIS}^{sem}\) and \(F_{IR}^{sem}\). To achieve cross-modal domain alignment in deep latent space, a series of residual blocks with fully shared weights \(W^{shared}\) is deployed at the bottleneck:

\[F_{VIS}^{shared} = F_{VIS}^{sem} + R(F_{VIS}^{sem}, W^{shared}), \quad F_{IR}^{shared} = F_{IR}^{sem} + R(F_{IR}^{sem}, W^{shared})\]

where \(R(\cdot, W^{shared})\) denotes the residual mapping parameterized by shared weights. This parameter-sharing scheme projects both modalities into a shared geometric manifold, providing a stable, modality-agnostic structural skeleton for downstream decoding.

2. Reverse-Pyramid Spectral Feature Encoder: Multi-Scale Modality-Specific Physical Mining

Distinct spectral domains possess unique physical mechanisms—such as color chrominance and surface reflection patterns in visible imagery, and thermal radiation highlights and temperature distributions in infrared imagery. To capture these exclusive characteristics without corrupting structural invariants, modality-specific spectral feature encoders \(En_{VIS}^{spec}\) and \(En_{IR}^{spec}\) are constructed with non-shared parameters. The encoders adopt an inverted pyramid convolutional hierarchy: shallow layers employ \(7\times 7\) kernels to capture broad contextual radiance statistics, intermediate layers utilize \(5\times 5\) convolutions to encode local texture patterns and temperature distributions, and deep layers utilize \(3\times 3\) convolutions for spatial compression. An adaptive global average pooling layer followed by dynamic projection compresses spatial features into low-dimensional modality-specific latent vectors \(F_{VIS}^{spec}\) and \(F_{IR}^{spec}\):

\[F_{VIS}^{spec} = En_{VIS}^{spec}(x), \quad F_{IR}^{spec} = En_{IR}^{spec}(y)\]

This hierarchical spectral modeling isolates exclusive physical properties, enabling the network to learn robust infrared thermal priors and visible reflectance embeddings independently.

3. Decoding-Reconstruction Module: Channel Concatenation and Dual-Branch Generation

Following the explicit decoupling of shared structures and modality-specific attributes, the decoding-reconstruction module recombines these heterogeneous vectors and translates them into high-dimensional target images. In the feature combination phase, the shared structural vector from the source domain is concatenated along the channel dimension with the target domain's specific spectral vector:

\[F_{IR}^{fused} = \text{Concat}(F_{VIS}^{shared}, F_{IR}^{spec}), \quad F_{VIS}^{fused} = \text{Concat}(F_{IR}^{shared}, F_{VIS}^{spec})\]

During unidirectional VIS-to-IR inference where no real infrared image is available, the target spectral feature vector \(F_{IR}^{spec}\) is derived from the learned thermal prior implicitly encoded within the trained network weights. In the feature reconstruction stage, the fused latent features are passed through multi-level residual blocks with instance normalization for nonlinear transformation, followed by transposed convolutions for spatial upsampling to recover full resolution:

\[\tilde{y} = G(x) = De_{IR}(F_{IR}^{fused}), \quad \tilde{x} = G(y) = De_{VIS}(F_{VIS}^{fused})\]

This decoupled combination mechanism guarantees that synthesized infrared images faithfully preserve the input visible geometry while rendering realistic, distinct infrared thermal signatures.

4. Semantic Structure Refinement: Discriminator-Driven Cross-Modal Alignment

Relying solely on shared residual weights within the encoder cannot theoretically guarantee optimal structural congruence across heterogeneous modalities. To overcome the limitations of passive residual alignment, a dedicated semantic structure discriminator \(D_{str}\) is introduced to enforce cross-modal semantic refinement via domain-adversarial learning. \(D_{str}\) is trained to distinguish whether a given shared feature originated from the visible or infrared branch, while \(En_{str}\) is optimized adversarially to fool \(D_{str}\), thereby compelling both branches to produce indistinguishable, domain-invariant representations:

\[\mathcal{L}_{sem} = \mathbb{E}_{x \sim p_{VIS}}\left[\log D_{str}(F_{VIS}^{shared})\right] + \mathbb{E}_{y \sim p_{IR}}\left[\log(1 - D_{str}(F_{IR}^{shared}))\right]\]

Minimizing this adversarial objective thoroughly strips residual modality-specific artifacts from \(F_{VIS}^{shared}\) and \(F_{IR}^{shared}\), aligning their latent distributions and preventing semantic drift during cross-spectral translation.

Loss & Training

The overall training objective combines the semantic structure adversarial loss \(\mathcal{L}_{sem}\), the bidirectional cycle consistency loss \(\mathcal{L}_{cyc}\), and the target domain adversarial loss \(\mathcal{L}_{adv}\) through weighted summation:

\[\mathcal{L} = \alpha \cdot \mathcal{L}_{sem} + \beta \cdot \mathcal{L}_{cyc} + \delta \cdot \mathcal{L}_{adv}\]

The cycle consistency loss \(\mathcal{L}_{cyc} = \mathbb{E}_{x}[\|P(G(x)) - x\|_1] + \mathbb{E}_{y}[\|P(G(y)) - y\|_1]\) guarantees bidirectional inversion stability under unpaired training scenarios. The domain adversarial losses \(\mathcal{L}_{adv}^x\) and \(\mathcal{L}_{adv}^y\) evaluated by discriminators \(D_X\) and \(D_Y\) ensure that generated images conform to target domain data distributions. Empirical ablation experiments demonstrate that setting \(\alpha=2, \beta=1, \delta=1\) yields optimal performance, proving that prioritizing semantic structural supervision is crucial for bridging the visible-infrared domain discrepancy. Training uses the Adam optimizer with an initial learning rate of \(1\times 10^{-4}\) and a batch size of 1 for 200 epochs, decaying the learning rate by 50% every 50 epochs on an RTX 4090 GPU.

Key Experimental Results

Main Results

Quantitative evaluations are conducted across the road-monitoring dataset AVIID, the forward-view driving dataset IRVI, and the self-collected aerial dataset DroneCoast. Experiments are repeated three times to compute mean and standard deviation across pixel-level (SSIM, PSNR), feature-level (FID, KID), and computational efficiency metrics:

Dataset	Metric	SHASP (Ours)	Prev. SOTA	Gain / Comparison
AVIID (Road Monitoring)	SSIM ↑	0.852 ± 0.005	0.840 ± 0.007 (PID)	+0.012 improvement, superior geometric fidelity
AVIID (Road Monitoring)	PSNR ↑	27.44 ± 0.21	27.04 ± 0.26 (DiffV2IR)	+0.40 dB higher, lowest signal distortion
AVIID (Road Monitoring)	FID ↓	41.08 ± 0.61	36.80 ± 0.95 (RGB-TIR)	Competitive second-best; outperforms general I2I models
IRVI (Driving View)	SSIM ↑	0.679 ± 0.002	0.671 ± 0.002 (StegoGAN)	Outperforms all GAN and diffusion baselines
IRVI (Driving View)	PSNR ↑	21.93 ± 0.21	21.68 ± 0.19 (StegoGAN)	Highest fidelity across dynamic driving scenes
IRVI (Driving View)	FID ↓	84.30 ± 0.87	89.11 ± 0.82 (Swin-UNIT)	-4.81 drop in feature distribution discrepancy
DroneCoast (Aerial Coastal)	SSIM ↑	0.760 ± 0.002	0.744 ± 0.001 (DiffV2IR)	Best contour preservation for small aerial targets
DroneCoast (Aerial Coastal)	PSNR ↑	24.89 ± 0.12	25.16 ± 0.17 (Swin-UNIT)	Competitive second-best, within 0.27 dB of top
DroneCoast (Aerial Coastal)	KID ↓	0.0379 ± 0.0009	0.0401 ± 0.0010 (StegoGAN)	Best local semantic and fine-grained feature modeling

In terms of computational complexity, SHASP contains 24.81M parameters and requires 50.7 ± 1.0 ms inference latency per \(256\times 256\) image. Compared to diffusion-based translation models such as DiffV2IR (162.20M, 185.9 ms) and PID (309.18M, 460.5 ms), SHASP delivers higher structural fidelity with roughly \(6.5\times\) to \(12\times\) fewer parameters and \(3.6\times\) to \(9\times\) faster runtime.

Ablation Study

Component ablations on AVIID and DroneCoast progressively evaluate the contributions of each module, alongside loss weight sensitivity:

Config	En_str	En_spec	Refine (L_sem)	AVIID SSIM ↑	AVIID FID ↓	DroneCoast SSIM ↑	DroneCoast FID ↓	Note
Model I (Baseline)				0.71	58.87	0.68	170.90	Vanilla CycleGAN baseline, severe blur
Model II	✓			0.77	50.76	0.72	153.25	Shared structure encoder yields +0.06 SSIM
Model III		✓		0.76	54.23	0.72	157.33	Spectral feature encoder improves physical cues
Model IV	✓	✓		0.80	45.39	0.74	136.95	Joint decoupling drops FID by 13.48
SHASP (Full Model)	✓	✓	✓	0.85	41.08	0.76	125.28	Adversarial refinement achieves top scores
SHASP (4× Data)	✓	✓	✓	0.86	40.15	0.77	124.09	Scales effectively with augmented training distributions

Loss weight coefficient ablations on the AVIID dataset: - Balanced control group (\(\alpha=1, \beta=1, \delta=1\)): SSIM of 0.846, FID of 43.77; - Strengthened semantic constraint (\(\alpha=2, \beta=1, \delta=1\)): SSIM peaks at 0.852, FID improves to 41.08 (optimal configuration); - Overweighting cycle consistency (\(\beta=2\)) or adversarial loss (\(\delta=2\)): SSIM drops to 0.844 and 0.835, while FID degrades to 42.29 and 41.62, respectively.

Key Findings

• Joint shared-specific decoupling provides the foundational leap: Progressing from the undecoupled baseline (Model I) to the dual-encoder decoupling configuration (Model IV) increases AVIID SSIM from 0.71 to 0.80 and lowers FID from 58.87 to 45.39, corroborating the architectural hypothesis that isolating shared structures from specific spectra is fundamentally superior to black-box style transfer.
• Adversarial semantic refinement completes the cross-modal bridge: Adding the structural discriminator \(D_{str}\) propels SSIM from 0.80 to 0.85 and FID from 45.39 to 41.08. t-SNE visualizations confirm that shared embeddings achieve tighter cross-modal clustering while modality-specific features remain distinctly separated.
• Synthesized data directly empowers downstream object detection: Training YOLOv9 on SHASP-synthesized infrared images substantially outperforms direct cross-domain inference with visible data, approaching the detection accuracy (AP50, Precision, and Recall) of models trained on real infrared data, thereby establishing high utility for data-starved downstream applications.

Highlights & Insights

• Representation decoupling over pixel-level mapping: By reframing cross-spectral synthesis as latent shared-specific disentanglement, SHASP leverages the physical insight that deep structural similarities overpower modal radiation divergences.
• Dual-tier alignment combining weight-sharing and domain discrimination: Coupling weight-shared bottleneck residual blocks with a dedicated adversarial semantic discriminator ensures thorough separation of modality-invariant geometry from modality-specific radiometry.
• Favorable fidelity-efficiency trade-off: Delivers visual and quantitative quality rivaling or exceeding multi-step diffusion pipelines while operating as a single-forward-pass GAN with only 24.81M parameters and ~50ms latency.

Limitations & Future Work

• Admitted limitations: The current validation is restricted to static cross-spectral image pairs and does not account for temporal coherence in video streams or extreme weather degradations (e.g., dense fog or total darkness).
• Noted limitations: During inference, infrared-specific attributes are drawn from static priors embedded in the trained weights rather than adaptively conditioned on external environmental variables (such as ambient temperature).
• Future directions: Incorporating explicit thermodynamic equations or sensor physical priors into the spectral encoder, and extending the shared-specific disentanglement paradigm to spatio-temporal video translation.

Related Work & Insights

• vs CycleGAN / CUT (Unpaired Image-to-Image Translation): CycleGAN and CUT rely on cycle consistency or patch contrastive losses without explicit representation decoupling, often generating severe geometric distortions and missing thermal hotspots; SHASP explicitly isolates structural invariants, preserving sharp object boundaries.
• vs DiffV2IR / PID (Diffusion-based IR Generation): PID and DiffV2IR use heavy diffusion reverse processes (160M–300M+ parameters, >180ms latency); SHASP achieves superior or competitive structural accuracy (SSIM/KID) with 85%+ lower computational overhead.
• vs RGB-TIR (Edge-Guided Translation): RGB-TIR relies on hand-crafted edge guidance which fails to model complex infrared radiation distributions; SHASP unifies structural alignment and multi-scale thermal feature extraction in an end-to-end framework.

Rating

• Novelty: ⭐⭐⭐⭐☆ Reframes cross-spectral translation into an elegant shared-specific decoupling and reconstruction paradigm with solid physical justification.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning three benchmark datasets, extensive quantitative metrics, component and hyperparameter ablations, t-SNE analysis, and downstream detection validation.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, well-motivated problem formulation, precise mathematical descriptions, and transparent qualitative comparisons.
• Value: ⭐⭐⭐⭐☆ Highly practical and efficient data augmentation framework for multimodal sensing and night-time vision tasks.