D3F-IR: Dual-Domain Deterministic Flow Matching for Visible-to-Infrared Translation

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/WanrenZeng/D3F-IR
Area: Image Generation
Keywords: Visible-to-Infrared, Flow Matching, Dual-Domain Decoupling, Deterministic Trajectory, Feature Modulation

TL;DR

To resolve the fundamental conflict between macroscopic thermal semantics and microscopic spatial detail reconstruction in visible-to-infrared translation, D3F-IR reformulates the generative process into dual-domain deterministic continuous flows—a latent semantic flow and an image-space pixel flow coordinated via patch-wise semantic modulation—delivering superior structural fidelity and physically plausible thermal radiation synthesis.

Background & Motivation

Visual-thermal sensor fusion is essential for robust perception in adverse environments such as autonomous driving and low-light surveillance, where standard RGB sensors suffer severe degradation under poor illumination while infrared sensors reliably capture thermal radiation. However, acquiring spatially calibrated and time-synchronized cross-modal paired datasets is prohibitively expensive and strictly constrained by specific hardware setups and viewpoints. Consequently, visible-to-infrared (V2IR) image translation has emerged as an attractive, scalable alternative to synthesize thermal training data from abundant visible-spectrum imagery. While early generative adversarial networks (GANs) suffered from training instability and severe geometric distortions, recent diffusion models and flow-based generative paradigms have advanced generation stability but still hit a critical performance plateau.

The core challenge stems from the fact that visible-to-infrared translation requires simultaneously inferring macroscopic physical thermal distributions and preserving microscopic spatial textures from RGB inputs that completely lack explicit temperature cues. Existing generative frameworks (such as PID, DiffV2IR, and ThermalGen) formulate the task as a unified generative process within a single continuous flow or a unified latent space. This formulation inevitably entangles global thermal semantic assignment with local spatial detail reconstruction. Forcing both heterogeneous objectives into a unified representation space severely compromises optimization, leading to blurred target boundaries, structural hallucinations, and unnatural thermal intensity artifacts (such as incorrectly assigning high thermal responses to painted white surfaces under shadows).

To eliminate this entangled compromise, the authors propose explicitly decomposing the cross-modal translation into two synchronized continuous flows across different abstraction spaces. Core idea: decouple visible-to-infrared translation into dual-domain deterministic flows (Dual-Domain Deterministic Flow Matching), where a semantic-domain flow driven by a Diffusion Transformer models macroscopic thermal radiation distributions, while a synchronized pixel-domain flow equipped with SEmantic-Aligned Modulation (SEAM) layers synthesizes fine-grained spatial details directly in image space.

Method

Overall Architecture

The D3F-IR framework formulates the translation from a visible image \(x_0\) to an infrared target \(x_1\) as two explicitly decoupled yet tightly coordinated continuous ODE paths: a semantic-domain flow running within the latent space of a fine-tuned DINOv3 encoder to capture macroscopic thermal distributions, and a pixel-domain flow operating directly in the high-resolution image space to construct fine-grained spatial geometry.

During inference, starting from the source image \(x_0\) and its initial semantic latent \(z_0 = \mathcal{E}(x_0)\), both states are updated synchronously across 10 discrete Euler steps. At each step, a Diffusion Transformer predicts the semantic velocity vector, which is converted via a lightweight cross-attention Fusion layer into a dynamic guidance signal \(s_t\). Conditioned on \(s_t\), a dedicated pixel-space velocity predictor estimates the instantaneous spatial velocity \(v^x_t\), advancing both latent and image states simultaneously until \(t=1\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Visible Image x0"] --> B["Semantic Latent Extraction<br/>DINOv3 encoder generates z0"]
    B --> C["Semantic-Domain Flow Evolution<br/>Diffusion Transformer predicts vzt"]
    C --> D["Cross-Domain Feature Fusion<br/>Cross-attention of z0 and vzt generates st"]
    D --> E["Pixel-Space Velocity Predictor<br/>Two-stage downsampling conv extracts p(0)"]
    E --> F["SEAM Patch-Wise Alignment Modulation<br/>Affine scaling, shift and spatial-channel attention"]
    F --> G["Dual-Domain Synchronous Euler Integration<br/>zt and xt advance synchronously to t=1"]
    G --> H["Output High-Fidelity Infrared Image x1"]

Key Designs

1. Dual-Domain Deterministic Flow Matching: Decoupling Thermal Semantics and Spatial Details To avoid the sub-optimal compromises incurred when optimizing temperature estimation and high-frequency spatial textures in a single space, D3F-IR models two continuous-time probability paths governed by deterministic ordinary differential equations (ODEs). Over the normalized time interval \(t \in [0, 1]\), a linear interpolation trajectory \(y_t = (1-t)y_0 + t y_1\) is adopted for both domains (\(y\) denoting latent \(z\) or image \(x\)), yielding a constant ground-truth target velocity field \((y_1 - y_0)\). The semantic flow endpoints \(z_0 = \mathcal{E}(x_0)\) and \(z_1 = \mathcal{E}(x_1)\) are defined by a trainable DINOv3 (ViT-S/16) backbone, with a Diffusion Transformer \(v^z_\theta(z_t, t \mid x_0)\) predicting the semantic velocity. Concurrently, the pixel flow connects endpoints \(x_0\) and \(x_1\) directly in image space, regressed by a pixel-space velocity predictor \(v^x_\phi(x_0 \mid s_t)\). By separating high-level thermal semantics from low-level spatial reconstruction, each flow optimizes its objective independently without destructive interference.

2. Cross-Domain Fusion and SEmantic-Aligned Modulation (SEAM): Establishing Explicit Local Correspondences Directly injecting semantic guidance into the pixel branch via global modulation (like AdaIN) lacks spatial granularity, whereas dense pixel-level modulation (like SPADE) or dense cross-attention often triggers spatial misalignment and severe memory overhead due to cross-modal domain discrepancies. To establish effective coordination, the predicted semantic velocity \(\hat{v}^z_t\) is first integrated with the source latent \(z_0\) using a cross-attention Fusion block to generate an informative guidance representation \(s_t\). Inside the convolutional pixel predictor, several SEAM layers iteratively modulate the pixel features. In the \(i\)-th SEAM layer, the guidance vector \(s^{(i)}\) passes through a Transformer block and a linear projection to yield affine scale \(\gamma^{(i)}\) and shift \(\beta^{(i)}\) parameters. These modulation parameters are broadcast patch-wise to corresponding \(4 \times 4\) regions of the downsampled pixel feature map \(p^{(i)}\): $\(p'^{(i)} = (1 + \gamma^{(i)}) \odot p^{(i)} + \beta^{(i)}\)$ The modulated features are subsequently refined by a spatial-channel attention block. This patch-wise alignment ensures that macro thermal constraints are strictly enforced across local structural regions while preserving fine spatial texture flexibility.

3. Dual-Path Teacher-Forcing Pixel Reconstruction Supervision: Stabilizing Velocity Learning Directly regressing flow matching velocity fields can suffer from loose spatial gradients during the early stages of training. To stabilize convergence and enhance high-frequency edge fidelity, D3F-IR establishes two supervision pathways for the pixel velocity predictor. The predicted path calculates \(\hat{v}^x_t\) using the estimated semantic guidance \(s_t\) to produce a one-step predicted image \(\hat{x}_1^{\text{pred}} = x_0 + \hat{v}^x_t\). Concurrently, the ground-truth path calculates \(\hat{v}^x_{\text{gt}}\) conditioned on the ground-truth target displacement \((z_1 - z_0)\) to form \(\hat{x}_1^{\text{gt}} = x_0 + \hat{v}^x_{\text{gt}}\). Both synthesized images are supervised against the target infrared image \(x_1\) using a combined reconstruction objective comprising \(\ell_1\), SSIM, and LPIPS losses: $\(\mathcal{L}_{\text{rec}}(\hat{x}_1, x_1) = \|\hat{x}_1 - x_1\|_1 + \lambda_{\text{ssim}}\bigl(1 - \text{SSIM}(\hat{x}_1, x_1)\bigr) + \lambda_{\text{lpips}}\mathcal{L}_{\text{LPIPS}}(\hat{x}_1, x_1)\)$ The ground-truth pathway acts as a teacher-forcing mechanism that provides stable, noise-free structural supervision, guaranteeing sharp object contours even when semantic predictions fluctuate.

Loss & Training

The entire architecture is trained end-to-end. The composite optimization loss balances the dual-domain flow regression loss and the dual-path image reconstruction loss: $\(\mathcal{L}_{\text{total}} = \lambda_{\text{flow}} \mathcal{L}_{\text{flow}} + \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}\)$ where the flow matching objective covers both semantic and pixel velocities: $\(\mathcal{L}_{\text{flow}} = \lambda_z \mathbb{E}_{t, z}\left[\|v^z_\theta(z_t, t \mid x_0) - (z_1 - z_0)\|^2\right] + \lambda_x \mathbb{E}_{t, x}\left[\|v^x_\phi(x_0 \mid s_t) - (x_1 - x_0)\|^2\right]\)$ Training uses the AdamW optimizer (\(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay = 0) with a Cosine Annealing with Warm Restarts scheduler (base learning rate \(4 \times 10^{-4}\), minimum \(4 \times 10^{-6}\), \(T_0 = 25, T_{\text{mult}} = 1\)). The semantic backbone uses DINOv3 ViT-S/16. Experiments are conducted on a single NVIDIA A100 GPU with a batch size of 1 and 8 gradient accumulation steps. From-scratch models train for 80 epochs, while the pre-trained variant is pre-trained on an augmented thermal dataset for 80 epochs and fine-tuned on target benchmarks for 20 epochs. Inference requires only 10 steps via an Euler ODE solver.

Key Experimental Results

Main Results

Quantitative evaluations are conducted across three standard benchmarks: KAIST, FLIR, and M3FD. D3F-IR is benchmarked against 13 representative approaches, including classical GAN models (Pix2Pix, CycleGAN, InfraGAN, DR-AVIT, StegoGAN, UNSB) and state-of-the-art diffusion/flow generative models (ControlNet, T2I-Adapter, F-ViTA, PID, ThermalGen, DiffV2IR). Evaluation metrics include PSNR, SSIM, LPIPS, and CLEAN-FID.

Dataset	Metric	D3F-IR (From Scratch)	D3F-IR (Pretrained†)	Strongest Baseline	Relative Performance / Note
KAIST	PSNR↑ SSIM↑ LPIPS↓ FID↓	24.938 0.830 0.246 75.909	24.738 0.834 0.219 40.312	23.743 (ThermalGen†) 0.801 (PID / ThermalGen†) 0.261 (ThermalGen†) 35.958 (ThermalGen†)	PSNR exceeds previous best by +1.195 dB; SSIM improves to 0.834; LPIPS drops to 0.219
FLIR	PSNR↑ SSIM↑ LPIPS↓ FID↓	20.094 0.541 0.360 66.716	19.581 0.542 0.268 60.964	19.596 (DiffV2IR†) 0.532 (PID) 0.309 (DiffV2IR†) 55.399 (DiffV2IR†)	From-scratch model surpasses all diffusion baselines in PSNR; pre-trained variant achieves 0.542 SSIM and 0.268 LPIPS
M3FD	PSNR↑ SSIM↑ LPIPS↓ FID↓	21.530 0.768 0.213 103.264	22.045 0.780 0.191 65.750	20.704 (DiffV2IR†) 0.744 (ThermalGen†) 0.214 (ThermalGen†) 63.154 (DiffV2IR†)	Pretrained PSNR surpasses 22.0 dB, SSIM gains +0.036, and LPIPS improves to 0.191

Ablation Study

1. Dual-Domain Architecture Ablation (Isolating Each Branch) Ablation experiments on FLIR and KAIST demonstrate the necessity of maintaining both branches:

Configuration	FLIR PSNR↑	FLIR SSIM↑	FLIR LPIPS↓	FLIR FID↓	KAIST PSNR↑	KAIST SSIM↑	KAIST LPIPS↓	KAIST FID↓	Note
Semantic Branch Only	18.938	0.530	0.375	74.542	24.692	0.828	0.253	87.316	Lacks direct pixel flow; produces blurred object contours and missing fine textures
Pixel Branch Only	17.542	0.504	0.487	163.088	22.079	0.762	0.338	126.868	Lacks global thermal priors; FID degrades severely, yielding visible-spectrum artifacts
Full Model (Ours)	20.094	0.542	0.360	66.716	24.938	0.830	0.246	75.909	Both domains synergize to achieve optimal fidelity and perceptual realism

2. Semantic Injection Mechanisms in SEAM Layers Comparing feature concatenation, full cross-attention, and patch-wise modulation:

| Injection Strategy | FLIR PSNR↑ | FLIR SSIM↑ | FLIR LPIPS↓ | FLIR FID↓ | KAIST PSNR↑ | KAIST SSIM↑ | KAIST LPIPS↓ | KAIST FID↓ | Note | |---|---|---|---|---|---|---|---|---| | Concatenation | 19.790 | 0.528 | 0.361 | 79.882 | 24.897 | 0.829 | 0.257 | 85.622 | Competitive distortion scores but weaker global distribution consistency (higher FID) | | Cross-Attention | 19.279 | 0.523 | 0.395 | 114.071 | 23.302 | 0.799 | 0.294 | 112.592 | Spatial cross-modal misalignment severely hurts FID and perceptual similarity | | Patch-wise Modulation (Ours) | 20.094 | 0.542 | 0.360 | 66.716 | 24.938 | 0.830 | 0.246 | 75.909 | Aligned local patch broadcasting stabilizes distributions and preserves spatial fidelity |

3. Comparison of Generative Modeling Paradigms Evaluating single-domain generative baselines against the dual-domain deterministic flow:

Generative Paradigm	FLIR PSNR↑	FLIR SSIM↑	FLIR FID↓	KAIST PSNR↑	KAIST SSIM↑	KAIST FID↓	Limitations & Characteristics
Diffusion (DDPM)	18.871	0.480	158.517	23.622	0.800	77.282	Struggles with trade-off; produces visible-spectrum artifacts
Flow Matching	18.896	0.516	152.909	22.993	0.796	104.149	Captures rough thermal trends but introduces structural warping
Deterministic Flow	18.938	0.530	74.542	24.692	0.828	87.316	Improves reconstruction but generates deviating thermal patterns
Dual-Domain Flow (Ours)	20.094	0.542	66.716	24.938	0.830	75.909	Decoupled coordination achieves top PSNR, SSIM, and FID scores

Key Findings

• Dual-Domain Decoupling Eliminates Trade-offs: Removing the pixel branch causes substantial boundary distortion, whereas eliminating the semantic branch leads to catastrophic degradation in thermal plausibility—evidenced by FLIR FID jumping from 66.716 to 163.088 as the model degenerates into a visible grayscale converter.
• Patch-Wise Modulation Outperforms Cross-Attention: Dense cross-attention suffers from feature misalignment between high-level semantic tokens and raw spatial pixels, causing FLIR FID to deteriorate to 114.071. Projecting semantic tokens to affine parameters broadcast over \(4 \times 4\) patches provides balanced regularization and superior fidelity.
• Superior Boundary Sharpness in 10-Step Generation: Qualitative inspections reveal that while CycleGAN, PID, and DiffV2IR blur pedestrian boundaries or incorrectly assign high thermal values to white paint in shadow, D3F-IR accurately distinguishes appearance cues from actual thermal radiation within just 10 Euler steps.

Highlights & Insights

• Dual-Domain Decoupled Flow Matching: Pioneers the separation of macroscopic physical thermal modeling and microscopic structural synthesis into two synchronized continuous ODE paths, preventing entangled optimization compromises.
• Efficient Deterministic Generation: Adopting straight deterministic flow trajectories minimizes sampling variance, preserves geometric integrity, and drastically reduces sampling steps down to 10 iterations without stochastic noise artifacts.
• Transferability of Patch-Wise SEAM Modulation: The SEAM module highlights that local affine parameter broadcasting can effectively bridge high-level semantic tokens with low-level spatial features without the fragility and computational cost of dense cross-attention.

Limitations & Future Work

• FID Dependency on Large-Scale Pretraining: Under the from-scratch setting, D3F-IR exhibits higher FID scores than baselines pre-trained on external large-scale datasets, indicating that distribution alignment is heavily constrained by small benchmark sample sizes.
• Physically Ill-Posed Scenarios Without Optical Cues: When visible inputs suffer complete texture loss (such as pitch-black environments or rain-soaked surfaces with zero contrast), inferring accurate thermal radiance without additional physical sensor priors remains ill-posed.
• Future Directions: Integrating explicit thermodynamic heat equation priors into the semantic ODE flow and incorporating test-time adaptation (TTC) to dynamically adjust thermal fields under adverse conditions.

Related Work & Insights

• vs PID (Pattern Recognition 2026): PID explores physics-informed diffusion in a unified latent space, but entangling features leads to warped object boundaries; D3F-IR achieves +1.63 dB higher PSNR on KAIST (24.938 vs 23.306) and delivers much cleaner structural contours.
• vs DiffV2IR (arXiv 2025): DiffV2IR incorporates vision-language priors into a diffusion pipeline but requires extensive denoising steps and produces over-smoothed edges; D3F-IR achieves 22.045 dB PSNR on M3FD within 10 ODE steps while maintaining sharp edge fidelity.
• vs ThermalGen (NeurIPS 2025): ThermalGen applies style-disentangled flow matching but often confuses visible light reflectivity with thermal emissivity under complex shadows; D3F-IR leverages DINOv3 semantic latents and SEAM layers to robustly decouple optical appearance from thermal radiance.

Rating

• Novelty: ⭐⭐⭐⭐☆ Decouples cross-modal translation into synchronized semantic and pixel deterministic continuous flows with patch-wise modulation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on three standard benchmarks against 13 baselines with detailed architectural, objective, and injection ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, mathematically rigorous, and articulates clear motivation regarding physical thermal modeling versus spatial detail synthesis.
• Value: ⭐⭐⭐⭐⭐ Provides an efficient, high-fidelity framework for multispectral dataset synthesis and autonomous driving perception with open-source code and low hardware overhead.