Skip to content

Physically Grounded Dual-Opacity Gaussian Splatting for Joint RGB-TIR Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Fulltext Cache: ../paper_cache/ECCV2026/eccv-5920.txt
Area: 3D Vision
Keywords: 3D Gaussian Splatting, RGB-TIR Reconstruction, Radiative Attribute Factorization, Dual-Opacity Rendering, Physics-grounded Rendering

TL;DR

RaViGS establishes a physically grounded 3D Gaussian Splatting framework for joint RGB-TIR scene reconstruction by parameterizing thermal response through intrinsic physical attributes (emissivity, temperature, reflectance) and decoupling cross-spectral visibilities via dual-opacity rendering, resolving fundamental radiative mismatches and occlusion conflicts on a shared geometric scaffold.

Background & Motivation

All-weather, round-the-clock 3D scene reconstruction is a foundational capability for autonomous systems, digital twins, and industrial inspection operating in degraded visual environments. While standard RGB-based neural representations excel at rendering intricate textural geometry and fine details, they fundamentally fail under adverse visibility conditions such as heavy smoke, total darkness, or deliberate camouflage. Thermal infrared (TIR) cameras observe long-wave radiation governed by material surface temperatures and thermal emissivity, providing vital observations that remain resilient against illumination failure. Jointly reconstructing 3D scenes from paired RGB and TIR imagery therefore promises both photorealistic visible novel view synthesis and metrically accurate thermal distribution fields.

However, joint RGB-TIR reconstruction faces a core tension rooted in conflicting image formation mechanisms and disparate spectral properties. Existing neural rendering approaches typically treat the thermal signal as an auxiliary single-channel grayscale intensity, optimizing it via direct numerical regression borrowed from visible color pipelines. This formulation violates radiative transfer physics (governed by the Stefan-Boltzmann law), where recorded thermal radiance is a non-linear superposition of material self-emission and reflected ambient radiation rather than passive diffuse reflection. Furthermore, RGB views feature dense, high-frequency textural gradients, whereas TIR images exhibit piecewise-smooth, low-frequency thermal distributions. Enforcing a single shared opacity field across both modalities forces high- and low-frequency gradients into direct conflict, compromising true color fidelity and introducing severe geometric blur.

Crucially, electromagnetic transmission and occlusion behavior diverge dramatically across spectrums. Common materials such as architectural glass are highly transparent to visible light (\(\alpha^{\text{rgb}} \to 0\)) yet essentially opaque and strongly absorbing in long-wave thermal infrared (\(\alpha^{\text{ir}} \to 1\)). Forcing a single visibility field across these modalities causes severe cross-spectral occlusion conflicts and ghosting artifacts along object boundaries. The angle of attack in this paper is to preserve a unified geometric anchor while decoupling the visual expression into intrinsic radiative attributes and modality-independent visibility fields. Core idea: develop a physically grounded dual-opacity Gaussian splatting framework (RaViGS) that factorizes thermal infrared radiance into emissivity, temperature, and reflectance, decouples modality-specific opacity fields to handle spectral transmission disparities, and coordinates bi-modal learning via co-support pruning and physics-constrained multi-objective optimization.

Method

Overall Architecture

RaViGS coordinates bi-modal RGB and TIR novel view synthesis on a unified 3D Gaussian primitive representation. Given multi-view synchronized and calibrated RGB-TIR image pairs initialized with COLMAP point clouds, each Gaussian primitive shares spatial geometric parameters (position \(\mu\), rotation \(R\), and scale \(S\)). The appearance and visibility modeling, however, splits into two decoupled branches: the RGB branch adopts standard spherical harmonics (SH) to capture view-dependent color with a dedicated visible opacity \(\alpha^{\text{rgb}}\), while the TIR branch implements Intrinsic Radiative Attribute Factorization with an independent thermal opacity \(\alpha^{\text{ir}}\). Both modalities undergo parallel differentiable \(\alpha\)-blending accumulation and backpropagate joint structural gradients into the shared geometric scaffold.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view RGB-TIR Image Pairs + Sparse Point Cloud"] --> B["Shared 3D Gaussian Geometric Scaffold<br/>Position / Rotation / Scaling"]
    B --> C["Intrinsic Radiative Attribute Factorization<br/>Factorized into Emissivity / Temperature / Reflectance"]
    B --> D["Spectrally Decoupled Dual-Opacity Rendering<br/>Decoupled RGB and TIR Transmittance Accumulation"]
    C --> E["Parallel Differentiable Rasterization<br/>RGB Color Field + TIR Radiative Energy Field"]
    D --> E
    E --> F["Physics-Constrained Joint Optimization<br/>Radiometric Reconstruction + Physical Prior + Edge-guided TV"]
    F --> G["Co-support Pruning Criterion<br/>Suppresses Single-modality Subsets & Preserves Unity"]

Key Designs

1. Intrinsic Radiative Attribute Factorization: physics-grounded thermal parameterization Addressing the fundamental flaw where regressing raw TIR intensity detaches neural primitives from thermodynamic principles and induces view-dependent temperature drift, RaViGS parameterizes each 3D Gaussian primitive with three physically grounded material properties: surface emissivity \(\epsilon\), normalized thermodynamic temperature \(T\), and environmental reflectance \(\rho\). To accommodate non-Lambertian emission, surface micro-facet shadowing, and directional thermal gradients, all three attributes are modeled via spherical harmonics (SH) evaluated along viewing direction \(d\) and constrained to \([0, 1]\) through Sigmoid activations. The synthesized infrared radiance \(E_i(d)\) strictly adheres to the Stefan-Boltzmann law and radiative equilibrium: $\(E_i(d) = \text{clip}\left(\epsilon_i(d) \cdot \sigma T_i(d)^4 + \rho_i(d) \cdot L_{\text{env}}, 0, 1\right)\)$ where \(T_i(d)^4\) captures the non-linear relationship between temperature and radiant emittance, \(\sigma\) is a learnable scaling factor absorbing the Stefan-Boltzmann constant and sensor response gain, and \(L_{\text{env}}\) represents ambient thermal irradiance. This decomposition forces the optimization to recover authentic thermodynamic properties instead of memorizing surface grayscale values.

2. Spectrally Decoupled Dual-Opacity Rendering: resolving cross-spectral visibility disparities To resolve geometric degradation and silhouette ghosting caused by sharp transmission discrepancies across wavelengths (e.g., glass showing visible transparency but infrared opacity), each Gaussian primitive maintains two distinct opacity values: \(\alpha^{\text{rgb}}\) and \(\alpha^{\text{ir}}\). Both branches share the identical depth-sorted Gaussian sequence \(\mathcal{N}(p)\) projected onto the image plane, but independently accumulate ray transmittances: $\(\hat{C}_{\text{rgb}}(p) = \sum_{i \in \mathcal{N}(p)} \mathcal{T}_i^{\text{rgb}}(p) \alpha_i^{\text{rgb}}(p) c_i(d), \quad \hat{I}_{\text{tir}}(p) = \sum_{i \in \mathcal{N}(p)} \mathcal{T}_i^{\text{ir}}(p) \alpha_i^{\text{ir}}(p) E_i(d)\)$ where the transmittance is accumulated as \(\mathcal{T}_i^m(p) = \prod_{j < i} (1 - \alpha_j^m(p))\) for \(m \in \{\text{rgb}, \text{ir}\}\). Shared geometric parameters receive joint multi-modal gradient updates (\(\lambda_{\text{rgb}} \frac{\partial \mathcal{L}_{\text{rgb}}}{\partial \text{geo}} + \lambda_{\text{ir}} \frac{\partial \mathcal{L}_{\text{ir}}}{\partial \text{geo}}\)), successfully isolating cross-spectral appearance discrepancies within the independent opacity and radiative parameters while fortifying geometric consistency.

3. Co-support Pruning Criterion: safeguarding unified multi-modal scaffolding To prevent the dual-opacity mechanism from degenerating into two disconnected Gaussian subsets that exclusively represent either the RGB or TIR spectrum, the framework establishes a co-support min-gating pruning criterion during adaptive density control. A primitive is marked for pruning only if its opacity falls below the pruning threshold \(\tau_\alpha = 0.005\) in both modalities simultaneously: $\(M_{\text{prune}}^{(k)}(i) = \mathbb{I}\left(\min\left(\alpha_i^{\text{rgb}}, \alpha_i^{\text{ir}}\right) < \tau_\alpha\right)\)$ This min-gating design guarantees that each retained Gaussian is anchored by shared multi-modal evidence. In challenging glass regions, weak optical cues (such as reflections or boundary highlights) enable low but non-zero \(\alpha^{\text{rgb}}\) values to survive pruning alongside high \(\alpha^{\text{ir}}\), faithfully preserving divergent cross-spectral visibility on a shared geometric structure.

Loss & Training

The framework is optimized end-to-end via a multi-objective loss function. The base photometric loss combines \(L_1\) and D-SSIM for each modality: \(\mathcal{L}_{\text{rec}}^m = \|I_m - I_m^{gt}\|_1 + \lambda_{\text{ssim}} \text{DSSIM}(I_m, I_m^{gt})\). To constrain the parameter space of the radiative attributes, a thermodynamic prior loss \(\mathcal{L}_{\text{phy}}\) grounded in Kirchhoff's radiation law is imposed: $\(\mathcal{L}_{\text{phy}} = \frac{1}{|\mathcal{K}|} \sum_{i \in \mathcal{K}} \left[ \mathcal{B}(\epsilon_i^0)^2 + \mathcal{B}(\rho_i^0)^2 + \text{ReLU}\left(\epsilon_i^0 + \rho_i^0 - 1 - \delta\right)^2 \right]\)$ where \(\mathcal{B}(x) = \text{ReLU}(-x) + \text{ReLU}(x - 1)\) bounds the base 0-th order SH components \(\epsilon_i^0, \rho_i^0 \in [0, 1]\), while enforcing energy conservation \(\epsilon^0 + \rho^0 \le 1 + \delta\) (\(\delta = 0.05\)). Additionally, an edge-aware smoothness loss \(\mathcal{L}_{\text{smooth}}\) leverages high-frequency RGB gradients to dynamically modulate TIR total variation regularization, relaxing smoothing near true structural edges while penalizing thermal noise in homogeneous regions. Combined with an SH coefficient regularizer \(\mathcal{L}_{\text{sh}}\) and an RGB gradient loss \(\mathcal{L}_{\text{grad}}\), the total objective is: $\(\mathcal{L}_{\text{total}} = \lambda_{\text{rgb}} \left(\mathcal{L}_{\text{rec}}^{\text{rgb}} + \lambda_g \mathcal{L}_{\text{grad}} + \lambda_{\text{sh}} \mathcal{L}_{\text{sh}}\right) + \lambda_{\text{ir}} \left(\mathcal{L}_{\text{rec}}^{\text{ir}} + \lambda_s \mathcal{L}_{\text{smooth}} + \lambda_p \mathcal{L}_{\text{phy}}\right)\)$ Optimization proceeds in two stages over 30,000 iterations: an initial 3,000-iteration geometric scaffolding warmup is followed by a non-uniform alternating optimization schedule that periodically freezes parameter subsets (covering 20% of training steps) to ensure stable opacity decoupling.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on the ThermoNeRF benchmark, comprising 16 indoor and outdoor scenes across varied temperature regimes. Metrics include PSNR, SSIM, and LPIPS for both modalities, alongside temperature metric Mean Absolute Error (MAE) and Region of Interest MAE (\(\text{MAE}_{\text{ROI}}\), in \(^\circ\text{C}\)).

Method PSNR (TIR) ↑ SSIM (TIR) ↑ LPIPS (TIR) ↓ \(\text{MAE}_{\text{ROI}}\) (\(^\circ\text{C}\)) ↓ MAE (\(^\circ\text{C}\)) ↓ PSNR (RGB) ↑ SSIM (RGB) ↑ LPIPS (RGB) ↓
Thermo-NeRF 31.101 0.943 0.153 1.550 0.603 19.971 0.622 0.273
Scaffold-GS - - - - - 20.077 0.625 0.340
Mip-Splatting - - - - - 20.536 0.664 0.345
MFTG 29.472 0.964 0.093 1.829 0.700 20.466 0.693 0.345
MSMG 28.155 0.962 0.116 1.924 0.788 20.551 0.708 0.383
OMMG 32.197 0.970 0.084 1.148 0.490 21.622 0.706 0.358
Thermal3DGS 28.397 0.958 0.081 2.335 1.041 - - -
3DGS+IR (Concatenation) 30.910 0.959 0.162 1.233 0.606 21.878 0.710 0.341
Ours (RaViGS) 32.479 0.975 0.068 1.425 0.472 22.011 0.713 0.337

Ablation Study

The table below validates the isolated contribution of each architectural component across all benchmark scenes:

Module Configuration (Dual-Opacity / 3-Physics / Loss-Constraint) TIR PSNR ↑ TIR SSIM ↑ TIR LPIPS ↓ \(\text{MAE}_{\text{ROI}}\) (\(^\circ\text{C}\)) ↓ TIR MAE ↓ RGB PSNR ↑ RGB SSIM ↑ RGB LPIPS ↓
Baseline (Vanilla Concatenation) 30.910 0.959 0.162 1.233 0.606 21.878 0.710 0.341
+ Dual-Opacity Only 32.068 0.969 0.075 1.194 0.517 21.358 0.693 0.359
+ 3-Physics Only 31.544 0.962 0.180 1.850 2.259 21.254 0.685 0.344
+ Loss-Constraint Only 31.422 0.966 0.118 1.132 0.522 21.080 0.691 0.344
Dual-Opacity + 3-Physics 31.567 0.962 0.071 1.334 0.528 21.316 0.696 0.340
3-Physics + Loss-Constraint 31.617 0.961 0.069 1.276 0.548 21.262 0.686 0.340
Ours (Full Model) 32.479 0.971 0.068 1.425 0.472 22.011 0.713 0.337

Key Findings

  1. Decoupled opacity is critical for suppressing boundary ghosting: Disabling the dual-opacity mechanism causes a sharp drop of approximately 1.5 dB in TIR PSNR, accompanied by noticeable silhouette ghosting. Decoupling visibility fields effectively resolves spectral occlusion mismatches under shared geometry.
  2. Stefan-Boltzmann factorization enforces reliable temperature inversion: Replacing the physical attribute parameterization with unconstrained direct intensity fitting causes temperature MAE to surge to 2.259 \(^\circ\text{C}\). Factorizing radiance into emissivity, temperature, and reflectance ensures radiometric convergence, achieving an overall temperature MAE of 0.472 \(^\circ\text{C}\).
  3. RGB gradient guidance sharpens thermal transitions: Filtering thermal noise via RGB-guided edge-aware smoothness transfers crisp geometric boundaries from visible textures to infrared fields without inducing over-smoothing.

Highlights & Insights

  • First-principles thermodynamic neural rendering: Transcends naive heuristic intensity regression by grounding 3D Gaussian radiance in the Stefan-Boltzmann law and Kirchhoff's radiation law, enabling verifiable and physically interpretable thermal property inversion.
  • Modality-decoupled visibility over shared geometry: Elegantly untangles the dilemma between rigid geometric sharing and cross-spectral transmittance disparities, providing a general blueprint for multi-modal rendering where media exhibit wavelength-dependent opacities.
  • Mutualistic co-support pruning: Enforces joint multi-modal existence via \(\min(\alpha^{\text{rgb}}, \alpha^{\text{ir}})\) thresholding, retaining challenging semi-transparent surfaces (like glass) through faint optical reflections while preventing independent point cloud drift.

Limitations & Future Work

  • Vulnerability to perfectly textureless transparent surfaces: The co-support pruning criterion assumes that semi-transparent objects exhibit non-zero RGB cues (specular glints or frame boundaries). In ideal cases devoid of any visible evidence, the optical opacity may degrade to zero and prematurely prune valid thermal primitives.
  • Steady-state thermal field assumption: The current formulation assumes thermal equilibrium and static ambient radiation, leaving dynamic transient thermodynamics (such as rapid convective cooling or shifting sunlight exposure) unmodeled.
  • Volumetric participating media unaddressed: While surface-level transmission disparities are resolved, complex volumetric absorption and scattering phenomena (e.g., dense aerosols, smoke plumes, or semi-absorbing liquids) require more sophisticated participating media radiation models.
  • Transitioning from implicit volumetric fields to explicit Gaussians: Early multi-modal novel view synthesis approaches (such as Thermo-NeRF) relied on dense volumetric neural fields with high query latency. RaViGS demonstrates that explicit 3D Gaussian primitives, when paired with physical radiative factorizations, provide superior temperature fidelity while preserving real-time rendering speeds.
  • Methodological implications for cross-spectral sensor fusion: Across autonomous driving LiDAR-camera setups, thermal surveillance, and multi-spectral satellite remote sensing, sensors often perceive heterogeneous aspects of the physical world. The paradigm of "shared geometric scaffold + decoupled channel visibility + physically grounded parameterization" provides a powerful template for broader multi-modal 3D perception tasks.

Rating

  • Novelty: 4.5 / 5.0 (Principled integration of thermal radiative physics with dual-opacity Gaussian splatting)
  • Experimental Thoroughness: 4.5 / 5.0 (Thorough evaluations spanning 16 diverse scenes, radiometric metric validations, and edge-case glass analysis)
  • Writing Quality: 4.5 / 5.0 (Clear formulation, elegant mathematical grounding, and comprehensive technical clarity)
  • Value: 4.5 / 5.0 (High practical value for all-weather autonomous navigation, infrastructure defect thermography, and robotics)