MTVDiff: Multimodal Conditional Latent Diffusion for Enhanced Thermal-to-Visible Face Translation

Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: thermal-to-visible face translation, multimodal diffusion model, cross-attention fusion, spatial feature transformation, identity preservation

TL;DR

MTVDiff tackles geometric discontinuities, semantic attribute mismatches, and identity degradation in thermal-to-visible face synthesis by integrating complementary depth maps and textual prompts into a frozen Stable Diffusion backbone, using a dual-branch cross-attention fusion module, gated text-to-visual feature alignment, and spatial feature transformations to achieve state-of-the-art visual quality and cross-modal face verification accuracy.

Background & Motivation

Cross-modal image translation across the thermal infrared and visible light spectra is a fundamental technology for security and surveillance applications, where thermal-to-visible (T2V) face translation enables robust identity verification in nighttime, adverse weather, and low-light environments. Because thermal sensors capture surface heat radiation rather than optical reflectance, thermal facial imagery lacks fine textures, chromatic attributes, and sharp geometric boundaries. Early efforts predominantly relied on Generative Adversarial Networks (GANs) such as Axial-GAN; however, adversarial training frequently suffers from mode collapse and optimization instability. Recent denoising diffusion probabilistic models, including T2V-DDPM and DiffTV, have improved perceptual realism and identity preservation, yet remaining single-modal or weakly conditioned formulations struggle to reconstruct faithful 3D facial topology and fine-grained attributes under severe lighting and pose variations.

The core tension lies in the severe cross-spectral domain gap: low-resolution thermal imagery inherently loses edge contrast and semantic cues, making it mathematically ill-posed to reconstruct high-fidelity visible faces with discriminative identity structures. While depth maps provide critical 3D geometric constraints and text descriptions offer high-level semantic specifications (e.g., gender, age, expression), conventional multi-condition paradigms like Uni-ControlNet fail to resolve spatial misalignments between modalities, resulting in entangled features, illumination artifacts, and degraded biometric verification.

To resolve these challenges, the authors investigate an explicit multimodal conditioning framework that leverages easily acquired auxiliary modalities—monocular depth maps providing 3D structural guidance and structured text prompts providing attribute control. Core idea: develop MTVDiff, a multimodal latent diffusion framework that dynamically fuses multi-scale thermal and depth features through a Dual-Branch Cross-Attention Fusion (DBCAF) module, aligns structured text embeddings via gated cross-attention, and injects the fused multimodal priors into frozen UNet residual blocks using Spatial Feature Transformations (SFT) to guarantee geometric fidelity and identity preservation.

Method

Overall Architecture

MTVDiff operates within the latent space of a pretrained Latent Diffusion Model (Stable Diffusion) while maintaining the 1.3B-parameter backbone fully frozen to safeguard its rich generative priors. The architecture processes three complementary modalities: thermal images as the primary identity guide, estimated depth maps for 3D geometric boundaries, and LLaVA-generated structured text descriptions for high-level semantic steering. Dual ResNet-18 encoders extract multi-scale features from thermal and depth inputs, which are processed by the Dual-Branch Cross-Attention Fusion (DBCAF) module to achieve time-step-aware bidirectional interaction. Concurrently, textual embeddings from CLIP are aligned with the fused visual representations via gated cross-attention. Finally, multi-scale multimodal features modulate the UNet residual blocks via Spatial Feature Transformations (SFT) during the reverse diffusion process, yielding sharp, identity-consistent visible faces.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Modalities<br/>Thermal Image + Depth Map + Structured Text"] --> B["Dual-Branch Cross-Attention Fusion DBCAF<br/>Multi-scale ResNet-18 + Temporal SFT Modulation + Cross-Attention"]
    C["Text Embedding Extraction<br/>Pretrained CLIP Text Encoder"] --> D["Gated Text-to-Visual Feature Alignment<br/>Gated Cross-Attention + Gated Self-Attention"]
    B --> D
    D --> E["Spatial Feature Transformation SFT Injection<br/>Predict affine scale/bias to modulate frozen UNet ResBlocks"]
    E --> F["Iterative Latent Denoising & VAE Decoding<br/>Synthesize high-fidelity visible facial image"]

Key Designs

1. Dual-Branch Cross-Attention Fusion: resolving thermal-depth cross-modal geometric misalignment Thermal images often exhibit blurred facial organ boundaries, while monocular depth maps offer distinct 3D structural layouts but may contain high-frequency sensor noise and lack photometric contrast. To reconcile both modalities across varying diffusion stages, the DBCAF module employs parallel ResNet-18 encoders to extract multi-scale feature hierarchies at resolutions of \(64\times 64\), \(32\times 32\), and \(16\times 16\). At each scale, a time-dependent MLP maps diffusion step \(t\) to scale, bias, and gating vectors: $\(s_i, b_i, g = \text{MLP}(t)\)$ Features are modulated by temporal SFT layers and then exchanged via bidirectional cross-attention: $\(Z_a = \text{softmax}\left(\frac{Q_b K_a^T}{\sqrt{\hat{C}}}\right) V_a\)$ where queries \(Q_b\) originate from modality \(b\), and keys/values \(K_a, V_a\) stem from modality \(a\). The concatenated representations pass through an MLP and Softmax to generate channel-wise dynamic blending weights \(w_a, w_b\). Following residual aggregation, a \(1\times 1\) convolution, feed-forward network, and gating modulation \(g\) adaptively calibrate structural emphasis across early and late denoising steps.

2. Gated Text-to-Visual Feature Alignment: suppressing semantic hallucinations and enhancing attribute controllability Injecting text prompts through unconstrained cross-attention in standard diffusion models easily overrides subtle thermal landmarks, producing hallucinatory facial textures and identity shifts; conversely, weak text conditioning fails to synthesize non-thermal facial traits such as hair styling, beard texture, and eye color. MTVDiff introduces a bounded gating mechanism where fused visual features \(H\) from DBCAF interact with CLIP text embeddings \(c^*\) via gated cross-attention (CA) and feed-forward layers: $\(H' = H + \lambda \cdot \tanh(\gamma_1) \cdot \text{CA}(H, c^*)\)$ $\(H^* = H' + \lambda \cdot \tanh(\gamma_2) \cdot \text{FFN}(H')\)$ where \(\lambda\) balances feature quality and semantic controllability, and \(\gamma_1, \gamma_2\) are learnable scalars. The \(\tanh(\cdot)\) activation naturally bounds the influence of textual semantics, preventing inaccurate or out-of-distribution prompts from distorting underlying geometric facial landmarks.

3. Spatial Feature Transformations: preserving pretrained generative priors and structural fidelity Direct end-to-end fine-tuning of large diffusion backbones risks catastrophic forgetting and heavy computational overhead. MTVDiff freezes the Stable Diffusion weights and incorporates lightweight SFT layers within UNet residual blocks across the downsampling hierarchy. In the \(l\)-th residual block, multi-scale fused features \(F_{\text{fusion}}^{(l)}\) produce spatial scaling and bias tensors via a \(1\times 1\) convolution: $\([\gamma_l, \beta_l] = \text{Conv}_{1\times 1}(F_{\text{fusion}}^{(l)})\)$ $\(h'_l = \text{GroupNorm}(h_l) \odot \gamma_l + \beta_l\)$ By directly transforming the GroupNorm normalized feature activations \(h_l\), the SFT mechanism seamlessly injects multi-modal identity and depth cues into the generative pipeline without disturbing the pretrained feature distribution, maintaining fine photometric textures while stabilizing facial verification performance.

Loss & Training

MTVDiff optimizes the standard latent denoising mean squared error objective: $\(\mathcal{L} = \mathbb{E}_{z_0, t, \epsilon \sim \mathcal{N}(0, I)} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c) \right\|_2^2 \right]\)$ where \(z_0 = \mathcal{E}(x_0)\) is the latent encoding of visible ground-truth faces, and condition \(c\) encapsulates the DBCAF and text representations injected via SFT. The 1.3B Stable Diffusion backbone is kept frozen, updating only 334M parameters (19.6% of the 1.7B total parameter budget). The framework is trained across \(4\times\) NVIDIA RTX 4090 GPUs using AdamW (learning rate \(5 \times 10^{-5}\), weight decay \(0.01\), batch size \(12\), 500 epochs, 1,000 linear noise schedule steps).

To prevent over-reliance on any single modality and ensure graceful degradation during deployment, an asymmetric modality dropout strategy is employed during training: depth and thermal modalities are independently dropped with probability \(p=0.1\), while text prompts are dropped with \(p=0.5\). This reflects real-world operational constraints where RGB-D hardware is reliably co-located, but text extraction requires auxiliary VLM inference.

Key Experimental Results

Main Results

Evaluation was conducted on two representative benchmarks: MCXFace (51 subjects, 6,120 image pairs across 3 illumination sessions) and SpeakingFaces (142 subjects, 7,700 synchronized thermal-visible pairs covering 9 viewing angles). Comparisons incorporate single-modal baselines (Axial-GAN, BBDM, AT-DDPM, T2V-DDPM) and multimodal baselines (DiffTV, DiffV2IR, Uni-ControlNet).

Table 1: Quantitative image synthesis performance on MCXFace and SpeakingFaces

Dataset	Methods	Modality Input	FID ↓	LPIPS ↓	PSNR ↑	SSIM ↑
MCXFace	Axial-GAN	Thermal	129.62	0.2131	17.82	0.6441
	BBDM	Thermal	127.06	0.1926	20.50	0.7629
	AT-DDPM	Thermal	123.57	0.2851	17.35	0.6344
	T2V-DDPM	Thermal	120.65	0.2445	18.37	0.6740
	DiffV2IR	Multimodal (Text+Seg)	79.33	0.1401	21.32	0.7671
	Uni-ControlNet	Multimodal (Th+Depth+Text)	97.01	0.2151	20.01	0.7181
	ControlNet	Thermal	86.13	0.2037	19.25	0.6797
	MTVDiff (Ours)	Multimodal (Th+Depth+Text)	75.33	0.1128	24.05	0.8355
SpeakingFaces	Axial-GAN	Thermal	46.69	0.1729	20.45	0.6673
	BBDM	Thermal	41.40	0.2112	26.51	0.7161
	AT-DDPM	Thermal	71.34	0.4063	8.75	0.5554
	T2V-DDPM	Thermal	36.33	0.2705	15.53	0.6832
	DiffTV	Multimodal (ID+Feat)	34.02	0.1650	29.45	0.7504
	DiffV2IR	Multimodal (Text+Seg)	27.79	0.3440	17.31	0.6432
	Uni-ControlNet	Multimodal (Th+Depth+Text)	29.20	0.3017	19.81	0.7042
	MTVDiff (Ours)	Multimodal (Th+Depth+Text)	14.37	0.1307	23.61	0.8623

Table 2: Biometric face verification performance using ArcFace feature embeddings

Dataset	Methods	Rank-1 ↑	VR@1% ↑	[email protected]% ↑
MCXFace	Axial-GAN	0.6815	0.0833	0.0033
	BBDM	0.7571	0.0960	0.0037
	AT-DDPM	0.5664	0.0704	0.0122
	T2V-DDPM	0.6815	0.1154	0.0256
	DiffV2IR	0.7834	0.1090	0.0192
	Uni-ControlNet	0.3758	0.0630	0.0126
	MTVDiff (Ours)	0.8726	0.2774	0.0258
SpeakingFaces	Axial-GAN	0.2357	0.1606	0.0402
	BBDM	0.8659	0.5933	0.1257
	AT-DDPM	0.4169	0.3422	0.0782
	T2V-DDPM	0.4940	0.3357	0.0816
	DiffTV	0.6885	0.0754	0.0159
	DiffV2IR	0.5159	0.3660	0.0679
	Uni-ControlNet	0.8619	0.6987	0.4555
	MTVDiff (Ours)	0.9376	0.8011	0.3131

Ablation Study

The ablation study systematically analyzes the contribution of the Depth Module (D), Text Prompt (T), and Cross-Attention in DBCAF (CA) across both datasets.

Table 3: Ablation study of MTVDiff components on MCXFace and SpeakingFaces

Variant	Depth (D)	Text (T)	Cross-Attn (CA)	MCXFace FID ↓	MCXFace LPIPS ↓	MCXFace SSIM ↑	SpeakingFaces FID ↓	SpeakingFaces LPIPS ↓	SpeakingFaces SSIM ↑
A	✗	✗	✗	85.79	0.1918	0.7232	20.14	0.1979	0.7721
B	✓	✗	✗	80.61	0.1226	0.8263	17.35	0.1382	0.8597
C	✗	✓	✗	86.13	0.1864	0.7335	19.16	0.1853	0.7768
D	✓	✗	✓	75.38	0.1132	0.8348	17.12	0.1362	0.8627
E	✓	✓	✗	78.10	0.1129	0.8292	15.22	0.1309	0.8681
Full Model	✓	✓	✓	75.33	0.1128	0.8355	14.37	0.1307	0.8623

Key Findings

• Depth geometry provides the foundational gain: Introducing monocular depth guidance (Variant B vs Variant A) accounts for the single largest metric jump, increasing SSIM from 0.7232 to 0.8263 and reducing LPIPS from 0.1918 to 0.1226 on MCXFace, proving that 3D structural boundaries stabilize facial organ layout.
• Cross-attention enables synergistic inter-modal alignment: Comparing Variant B (depth without CA) and Variant D (depth with CA) isolates the role of bidirectional cross-attention in DBCAF, driving a 6.5% FID reduction on MCXFace (80.61 to 75.38) and a 7.6% LPIPS reduction (0.1226 to 0.1132).
• Text semantics require geometric grounding: Relying solely on text conditioning (Variant C) fails to improve FID over the baseline (86.13 vs 85.79 on MCXFace) due to hallucinated facial boundaries. However, once depth constraints are active (Variant E vs Variant B), text prompts successfully enhance perceptual realism and reduce FID to 78.10.
• Dynamic stage-dependent parameter trajectory: Tracking the temporal gating trajectory confirms that scale parameters (\(s_1, s_3\)) peak during early diffusion steps (\(t=800-1000\), reaching \(0.676\) and \(0.732\)) to establish coarse geometry, while decaying toward final steps (\(t=0-200\), down to \(0.589\) and \(0.633\)) as bias parameters become active to preserve fine identity cues.
• Robust performance under modality dropout: In extreme inference conditions where depth and text are both removed, MTVDiff* still obtains an FID of 14.69 on SpeakingFaces, outperforming DiffTV (34.02), demonstrating that the learned architecture provides superior inductive bias independent of auxiliary inputs.

Highlights & Insights

• Decoupled geometric and semantic conditioning: Separates 3D spatial priors (depth via DBCAF and SFT) from high-level descriptive priors (text via bounded gated cross-attention), avoiding feature entanglement and cross-spectral hallucinations.
• Diffusion-timestep-aware dynamic modulation: Uncovers and models the temporal shifts in conditional requirements across diffusion steps, replacing heuristic manual stage divisions with continuous MLP-driven modulation.
• Parameter-efficient adaptation: Freezes 80.4% of model weights (the 1.3B Stable Diffusion backbone), proving that high-fidelity cross-spectral domain transfer can be achieved by training only a lightweight 334M conditional adapter.

Limitations & Future Work

• Inference dependency on vision-language models: Text conditioning relies on prompt generation from LLaVA at test time; prompt analysis reveals that irrelevant or inaccurate descriptions degrade generation performance (FID worsens by 13.2%).
• Fixed spatial resolution: The current implementation operates strictly at \(256\times 256\) resolution; adapting to high-definition face synthesis (\(512\times 512\) or higher) will require scaling the DBCAF receptive field.
• Degradation under total modality loss: Performance still degrades measurably when both depth and text signals are unavailable during inference, indicating the need for self-supervised geometric completion from single thermal inputs.

Related Work & Insights

• vs Uni-ControlNet: Uni-ControlNet is a generic multi-condition framework lacking thermal-specific identity and geometry alignment, yielding significant chromatic artifacts and lower verification accuracy (0.3758 vs 0.8726 on MCXFace); MTVDiff establishes strict spatial and biometric fidelity.
• vs DiffTV: DiffTV relies on 2D feature alignment and ArcFace latent features without explicit 3D geometry; MTVDiff incorporates monocular depth maps to resolve pose and view variations, outperforming DiffTV in Rank-1 and [email protected]%.
• vs DiffV2IR / T2V-DDPM: Earlier pixel-space diffusion methods require heavy iterative computation and rely on coarse semantic segmentations; MTVDiff leverages pretrained latent diffusion priors to achieve higher perceptual realism and faster convergence.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegant decoupled multimodal framework featuring DBCAF and dynamic temporal gating.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-dataset evaluations, ArcFace verification at strict thresholds, ablation studies, and prompt sensitivity analyses.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically rigorous, with clean diagrams and consistent notations.
• Value: ⭐⭐⭐⭐☆ Highly applicable to surveillance, low-light face recognition, and cross-spectral biometrics.