AV2T-Gen: Aerial Visible to Thermal Generation with Environment and Vehicle State Guidance¶

Conference: ECCV2026
Paper: ECCV Official Page · PDF
Code: https://github.com/NPU-CVPG/AV2T
Area: Autonomous Driving / Aerial Cross-Modal Image Generation
Keywords: Visible-to-thermal translation, environmental conditions, vehicle motion states, foundation-model guidance, latent diffusion

TL;DR¶

AV2T-Gen adds environmental prompts and motion-aware object conditions to Instruct-Pix2Pix-style diffusion so that aerial RGB-to-thermal translation goes beyond appearance transfer, reducing ThermalGen's FID from 83.16 to 67.45 in joint four-scene evaluation while improving vehicle detection on synthetic thermal images.

Background & Motivation¶

Visible and thermal images are not simply two color palettes for the same scene. RGB appearance largely reflects visible illumination and reflectance, whereas thermal appearance also depends on object temperature, material properties, and atmospheric conditions. Two visually similar vehicles can therefore exhibit different thermal signatures because of their motion states or environments. Treating RGB-T translation as style transfer can produce plausible outlines with contextually inappropriate thermal patterns. Meanwhile, synchronized and calibrated aerial RGB-T pairs are expensive to collect, and synthetic pairs can support cross-modal matching and object detection, making translation quality practically important.

GAN and diffusion approaches have progressively improved sharpness and structural preservation. F-ViTA additionally uses foundation models such as SAM, Grounding DINO, and CLIP for semantic and boundary guidance. Yet knowing that an object is a vehicle does not establish its expected thermal behavior at capture time, and a single frame rarely determines motion reliably. The authors consequently introduce two external sources of evidence: vehicle states obtained from aerial sequences, and temperature, humidity, pressure, wind speed, and related environmental information associated with capture location and time. They also build a paired dataset carrying these metadata.

The objective is to reduce missing conditioning information in RGB-to-thermal translation, not to replace a complete thermal physics model with a generator. Core idea: describe global imaging conditions through environmental text, locate local thermal attributes through object masks and motion-aware labels, and use both to condition latent diffusion denoising.

Method¶

Overall Architecture¶

Inputs comprise a visible image, vehicle-state annotations, and environmental information; the output is its corresponding thermal image. Training additionally requires a paired real thermal image for supervision. Deployment does not require that thermal target, but obtaining vehicle states depends on sequence analysis and obtaining environmental conditions requires metadata, so this is not a strictly single-frame RGB-only system.

The pipeline consists of condition acquisition, contextual fusion, and conditional denoising. A frozen VAE encodes the visible image as a structural condition and the real thermal image as the training target. CLIP encodes environmental prompts, while a trainable MLP combines object-label and SAM-mask embeddings. Together, these conditions guide the UNet to generate a thermal latent representation, which the VAE decodes into an image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Aerial RGB sequence<br/>Location and capture time"] --> B["Environment and State Acquisition"]
    B --> C["Mask and Semantic Fusion"]
    B -->|Environmental prompt via CLIP| D["Conditional Latent Denoising"]
    C -->|Object context| D
    A -->|RGB via frozen VAE| D
    N["Noisy thermal latent<br/>From real images during training"] --> D
    D --> E["Frozen VAE decoding<br/>Generated thermal image"]

Key Designs¶

1. Environment and State Acquisition: supply conditions that a single RGB frame cannot determine

Vehicle states are not arbitrary prompt choices. The authors first train an aerial-view YOLOv11 visible-light vehicle detector, then use temporal sequences to determine whether the same vehicle moves, producing stationary, moving, or unknown labels. The unknown category preserves indeterminate cases. However, the main paper delegates the detailed algorithm to supplementary material and does not specify the complete track-association procedure, camera-motion compensation, or movement threshold. These should not be filled in as known implementation details. Motion provides additional evidence for thermal appearance rather than directly measuring engine temperature, so the state label can still diverge from the actual thermal state.

The environmental branch uses GPS and capture time to obtain time, temperature, humidity, pressure, and wind speed from an open climate platform. Discrete observations are interpolated into continuous values and assembled into a text prompt, which CLIP encodes. This text interface fits the existing image-editing diffusion model, but the meteorological quantities remain learned conditions rather than parameters inserted into a radiative transfer equation. The authors interpret humidity and pressure as proxies for atmospheric attenuation and semantic grounding as a prior for material emissivity. These interpretations are plausible, but they do not establish a calibrated physical solver.

2. Mask and Semantic Fusion: associate vehicle states with the correct object regions

A global prompt saying that moving vehicles are present does not specify which vehicle should receive the corresponding thermal attributes. AV2T-Gen uses Ram-Grounding to obtain generic object labels and bounding boxes, then merges them with automatic vehicle-state annotations to form complete labels and corresponding locations. SAM processes these boxes to produce object masks. The masks provide boundaries and spatial localization, while CLIP-encoded complete labels provide object semantics and state information. This context can include background objects such as buildings as well as vehicles; it is not restricted to vehicle detection boxes.

Mask and label embeddings are concatenated and passed through a trainable MLP to produce joint context. This branch and the environmental prompt embeddings jointly condition generation. Equation (3) describes their organization as:

\[ E_c = \bigl(E_p,\operatorname{MLP}(E_m,E_l)\bigr). \]

Here, \(E_p\) denotes environmental prompt embeddings, \(E_m\) mask embeddings, and \(E_l\) label embeddings that include state information. The parentheses indicate jointly supplied conditions: the main paper does not establish whether the final fusion uses elementwise addition or a particular token-concatenation layout, nor does it specify the full mask-encoder dimensions. This note does not invent those details. The important mechanism is preserving the association among object identity, location, and state, allowing global environmental context and local object conditions to play distinct roles.

3. Conditional Latent Denoising: learn thermal appearance while preserving RGB structure

The backbone follows the VAE and diffusion approach of Instruct-Pix2Pix. A frozen VAE encoder compresses the visible input into \(I_E\), retaining scene layout and object shape. The real thermal image is encoded into \(Z\), to which Gaussian noise is added during training. The noisy thermal latent and \(I_E\) are concatenated as the UNet input, while joint context \(E_c\) conditions denoising. QKV attention within the UNet relates spatial structure to textual semantics. The resulting thermal latent is decoded by the frozen decoder, so the RGB structural constraint and the thermal output distribution are not implemented by directly sharing pixel values.

Masks and label semantics can therefore constrain generation throughout the denoising process rather than merely correcting the final output. At inference, a random thermal latent is denoised under RGB and contextual conditions; the real thermal image used in training must not be mistaken for an inference input. Equation (4) is only a shorthand for the denoising mapping and does not establish a sampler, noise schedule, step count, or exact attention tensor arrangement. It does not justify claiming a specific DDIM configuration. Foundation models remain frozen, and the authors identify the MLP and UNet as trainable components; total system parameters must not be equated with trainable parameters.

A Worked Example¶

Consider an aerial sequence showing moving and stationary vehicles in the same parking area. This is an explanatory example, not an additional experiment. Sequence analysis first assigns moving or stationary labels to individual vehicles, retaining unknown where the state cannot be determined. Temperature, humidity, pressure, and wind speed associated with location and capture time form a shared environmental prompt. Ram-Grounding and state annotations provide labels and boxes, and SAM separates the corresponding object regions.

The MLP receives these regions and their associated semantics. The UNet sees global meteorological context, per-object states, and the RGB layout together, producing a spatially corresponding thermal image. Without states, it must infer vehicle thermal attributes from appearance alone. Without environmental prompts, similar appearances under different conditions may collapse toward average thermal patterns. The two ablations test whether these conditions provide distinct benefits; the example does not assert a physical law that moving vehicles must always be hotter.

Loss & Training¶

Supervision includes an L1 loss between predicted and real thermal latents, an edge L1 loss after applying Sobel to generated and real images, and a VGG perceptual loss. Latent supervision constrains the overall conversion, the edge term discourages degradation of scene structure and vehicle boundaries, and the perceptual term supports image-level detail. Following the three terms and weights explicitly described in the text, the objective can be organized as:

\[ \mathcal{L}=1.0\mathcal{L}_{\mathrm{Latent}}+0.5\mathcal{L}_{\mathrm{Edge}}+0.25\mathcal{L}_{\mathrm{perception}}. \]

The cached extraction of equation (5) has a damaged connector and coefficient marker before the third term. The expression above is reconstructed from the three terms and weights explicitly stated in the adjacent prose, not copied character for character; refer to the original paper for exact typesetting. The main text does not specify the VGG layers, feature normalization, or optimizer settings, so these are not invented. Training runs for 100 epochs on 8 RTX 4090 GPUs with 24GB each, using \(640\times512\) images for both training and testing.

Key Experimental Results¶

Main Results¶

The paired dataset covers four aerial scenes: a substation, a school, a square, and a driving school. It contains 29,672 image pairs, with each scene split 9:1. The prose calls this training/validation, whereas Table 2 calls it training/test; this note uses the neutral term evaluation split. The following reproduces the All scenes columns of Table 1, representing joint training and evaluation across the four scenes, not held-out-scene transfer. Higher PSNR, measured in dB, and SSIM are better; lower FID and LPIPS are better.

Method	PSNR	SSIM	FID	LPIPS
DiffV2IR	22.35	0.620	108.54	0.39
PID	22.33	0.621	106.13	0.36
sRGB-TIR	22.34	0.619	110.83	0.39
ThermalGen	23.94	0.734	83.16	0.37
AV2T-Gen	24.08	0.750	67.45	0.29

Relative to ThermalGen, PSNR improves by only 0.14 dB, whereas FID decreases by 15.71 and LPIPS by 0.08. The gains are more pronounced in perceptual and distributional metrics. Identical data splits do not imply identical information budgets: AV2T-Gen additionally uses environmental and state metadata. The comparison supports the utility of a system with these conditions, not a strictly isolated backbone comparison.

Ablation Study¶

The following comes from Table 7. All configurations retain the latent loss, and each ablation removes only its named component; the removal order agrees with the description of Figure 6.

Configuration	PSNR	SSIM	FID	LPIPS
Without perceptual loss	23.14	0.716	72.19	0.30
Without edge loss	22.90	0.722	68.00	0.30
Without vehicle states	22.93	0.732	73.83	0.30
Without environmental prompts	22.29	0.700	77.66	0.31
Full model	24.08	0.750	67.45	0.29

Removing environmental prompts reduces PSNR by 1.79 dB and increases FID by 10.21, the largest effect among these removals. Removing vehicle states reduces PSNR by 1.15 dB and increases FID by 6.38. Edge loss affects PSNR more noticeably than FID, indicating that the terms address different aspects of quality. Table 8 further removes individual weather variables; the authors rank temperature, humidity, wind speed, and pressure in decreasing importance. This does not establish the causal identifiability of their individual effects.

Downstream Evaluation and Generalization¶

Table 9 applies the same detector to thermal images generated by each method. The following selects the comparison with ThermalGen; all metrics are percentages.

Method	mAP	mAP50	mAP75	Small-object mAP
ThermalGen	11.7	29.4	6.9	10.7
AV2T-Gen	15.6	35.9	11.1	14.2

The mAP gain is 3.9 percentage points, showing that improved synthesis also affects features usable by a detector. This evaluates a detector trained on real thermal data and tested on synthetic images; it must not be conflated with the real-domain benefit of synthetic training data. Setup B in Table 10 instead trains on synthetic data and tests on real thermal images, reporting AV2T-Gen moving/static mAP50 of 78.4/73.1.

For LLVIP cross-dataset evaluation in Table 4, zero-shot PSNR is 13.82/12.58 for AV2T-Gen/ThermalGen, increasing to 23.68/23.12 after fine-tuning. In leave-one-scene-out evaluation, AV2T-Gen loses 1.18 to 1.52 dB PSNR relative to the corresponding in-domain results. This supports some transferability, not lossless generalization. The visible rows of Table 4 do not contain F-ViTA, so its numerical comparison cannot be supplied from the prose alone.

Highlights & Insights¶

The conditions target information genuinely missing from cross-modal translation rather than simply adding network layers. Sequences provide motion evidence and meteorological metadata provide global context, introducing learnable explanatory variables for similar visible appearances with different thermal signatures.
Pairing masks with state labels provides clearer object attribution than an isolated global prompt. This could transfer to other cross-modal generation tasks requiring per-object attribute control, but propagation of mask errors and uncertain labels still needs evaluation.
Ablations, cross-scene transfer, and downstream detection provide different kinds of evidence. They respectively test whether conditions help, whether performance depends on fixed scenes, and whether generated images have task utility; a single FID result cannot answer all three questions.

Limitations & Future Work¶

Physical realism is supported indirectly. The authors explicitly do not solve complete radiative transfer equations, and the main text does not report absolute temperature errors or radiometric calibration results. PSNR, SSIM, FID, LPIPS, and detection metrics cannot establish temperature accuracy; controlled environmental interventions and radiometric measurements would strengthen the evaluation.
Input conditions can be inaccurate. Motion is not engine thermal state, and residual heat after parking, occlusion, and camera motion can affect the correspondence. The main text does not provide the complete state-estimation algorithm. The authors propose incorporating real-time sensors and finer-grained tracking history in future work.
Data splitting and fairness have boundaries. Continuous sequences use a 9:1 split, but the main text does not clarify whether capture-time blocks are isolated, leaving possible adjacent-frame correlations unresolved. Additional metadata and frozen foundation models also constitute system-resource advantages. Leave-one-scene-out and LLVIP experiments help without fully settling these issues.
Inference is expensive and the hardware description is inconsistent. The complete system has approximately 15.5B parameters and takes about 10.00 seconds per image, of which diffusion accounts for 9.45 seconds. The prose attributes the total latency to RTX 4090, while Table 3 specifies RTX L40; these cannot be treated as a single consistent hardware measurement. The intended application is offline data generation, not real-time onboard inference.
The available cache and reporting have gaps. Supplementary material is not included in this cache, the extracted loss equation is damaged, and some baseline descriptions do not fully match the tables. This note retains verifiable results without inventing sampling hyperparameters, missing baselines, or unspecified implementation details.

Compared with Instruct-Pix2Pix: AV2T-Gen adopts its latent image-editing framework but specializes the conditions to environmental context and vehicle states needed for thermal generation. Its primary contribution lies in conditioning and data rather than an entirely new diffusion backbone.
Compared with F-ViTA: Both exploit foundation models for semantic and spatial guidance, while AV2T-Gen emphasizes explicit environmental variables and per-vehicle states. The available cache provides no tabulated F-ViTA numerical results, so this is a methodological comparison, not a claim of quantitatively established superiority.
Compared with PID and ThermalGen: PID emphasizes physically relevant constraints, and ThermalGen uses style-disentangled flow-based generation. AV2T-Gen instead introduces observable external conditions into diffusion. Comparisons with identical metadata access and training budgets would better separate the contributions of the backbone and the conditioning information.

Rating¶

Novelty: 4/5. Joint control through environmental metadata and per-vehicle states addresses a specific task need, while the backbone and foundation components largely reuse existing methods.
Experimental Thoroughness: 4/5. Main comparisons, component ablations, cross-dataset and held-out-scene tests, and detection evaluations are provided, but radiometric calibration and information-budget fairness remain incomplete.
Writing Quality: 3/5. The overall pipeline is understandable, but hardware reporting, baseline descriptions, and some implementation details need clarification.
Value: 4/5. Useful for offline aerial RGB-T dataset construction, but not a replacement for real thermal sensors or a directly deployable real-time system.