ThermoGS: Decoupling Physical Surface Attributes for Spatio-Temporal Thermal Field Emulation via 4D Gaussian Splatting¶
Conference: ECCV 2026
Paper: CVF Open Access
Code: https://github.com/NPU-CVPG/ThermoGS
Area: 3D Vision
Keywords: Thermal Simulation, 4D Gaussian Splatting, Thermodynamics, Physics-Informed Neural Networks, Physical Attribute Decoupling
TL;DR¶
ThermoGS integrates 4D Gaussian Splatting with thermodynamic energy balance laws and PINNs to decouple intrinsic surface properties (emissivity, convective heat transfer coefficient, volumetric heat capacity, and absorptivity), achieving physically-consistent all-weather spatio-temporal thermal field emulation with temperature prediction error under 1°C.
Background & Motivation¶
Thermal infrared (TIR) imaging provides an indispensable capability for all-weather perception by directly measuring surface radiative temperature through adverse conditions such as darkness and haze. These distinct advantages have catalyzed widespread applications in intelligent security surveillance, building energy auditing, and large-scale urban infrastructure inspection. With the advent of high-resolution, lightweight thermal sensors and autonomous drone platforms, the foundation for capturing large-scale outdoor thermal environments has been firmly established. However, conventional 3D thermal processing pipelines remain constrained within a reconstruction-centric paradigm: they first construct geometric surface meshes using visible RGB multi-view stereo (MVS) and subsequently project pre-captured thermal textures onto these static geometries. Such approaches function merely as passive data-driven snapshot replicas, incapable of simulating thermal dynamics or extrapolating temperature evolutions when ambient environmental parameters—such as solar irradiance, wind speed, or humidity—fluctuate.
Recent advances in neural rendering, notably NeRF and 3D Gaussian Splatting (3DGS), have generalized thermal modeling into novel-view synthesis tasks, as demonstrated by frameworks like ThermoNeRF and Thermal3D-GS. Nevertheless, most existing architectures directly regress view-dependent thermal radiance without modeling the underlying heat transfer mechanics. Even temporal formulations such as NTR-Gaussian are restricted to nighttime reconstructions under static environmental conditions, completely lacking the physical mechanisms required to accommodate daytime solar radiation. Crucially, prior methods fail to decouple the intrinsic thermo-physical properties of surfaces (such as heat capacity, convective coefficients, emissivity, and solar absorptivity) from transient temperature states. Consequently, these models suffer from severe geometric instability, lack interpretability, and break down when predicting thermal responses across unseen meteorological scenarios.
This work argues that the ultimate objective of 3D thermal processing should transcend passive geometric reconstruction into active, deep-learning-based physical scene emulation. Core idea: couple the thermodynamic first-law energy conservation equations with 4D Gaussian Splatting using Physics-Informed Neural Networks (PINNs) to explicitly decouple intrinsic surface emissivity, convective transfer coefficient, volumetric heat capacity, and solar absorptivity on Gaussian ellipsoid scales, realizing physically-consistent, all-weather continuous thermal field simulation.
Method¶
Overall Architecture¶
The ThermoGS pipeline begins with an explicit 3D Gaussian representation constructed from drone oblique visible-light photography. By combining 3D Gaussian spatial features, Fourier positional encodings of time, and synchronized meteorological measurements (ambient temperature, relative humidity, atmospheric pressure, wind speed, and solar irradiance), a physics-informed neural network predicts both the transient absolute surface temperature \(T\) and intrinsic physical properties (\(e, h, c, \alpha\)) for each individual Gaussian ellipsoid. The thermodynamic heat transfer and radiative partial differential equations (PDEs) are analytically grounded onto the geometric scale of the 3D Gaussian ellipsoids. Supervised jointly by radiometric-corrected multi-view thermal images and rigorous physical conservation laws, ThermoGS enables high-fidelity thermal rendering and temperature simulation from arbitrary viewpoints, across extended time periods, and under synthetic climate perturbations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs<br/>Geometry + Spatial-Temporal Encoding + Climate Data"] --> B["Physical Attribute Decoupling Network<br/>Multi-head Residual MLP for Temperature and Properties"]
B --> C["Gaussian-Scale Thermodynamic Coupling<br/>Ellipsoid Surface Area and Volume in Energy Balance"]
C --> D["Spatio-Temporal Physics-Constrained Optimization<br/>PDE Supervision + ODE Trajectory Integration"]
D --> E["High-Fidelity Thermal Simulation<br/>Arbitrary-View Temperature and Property Maps"]
Key Designs¶
1. Gaussian-Scale Thermodynamic Energy Balance Coupling: Grounding Physical Conservation on Explicit Geometric Primitives
Outdoor surface temperature dynamics under clear-sky conditions are governed by the instantaneous energy exchange balance between the object surface and its surrounding atmosphere. During daytime, objects absorb solar radiation while continuously dissipating heat via thermal radiation and convective transfer; at night, radiative cooling and convective dissipation dominate. The continuous governing partial differential equation (PDE) over time \(t\) is formulated as: $\(c V \frac{\partial T}{\partial t} = Q_{absorb} - Q_{rad} - Q_{conv}\)$ where solar heat absorption is determined by material absorptivity \(\alpha\), solar projected surface area \(A_{\perp}\), and solar irradiance \(G_{sun}\) (\(Q_{absorb} = \alpha A_{\perp} G_{sun}\)); thermal radiation release follows the Stefan-Boltzmann law \(Q_{rad} = e \sigma A_{rad} T^4\) (\(e\) is surface emissivity, \(\sigma\) is the Stefan-Boltzmann constant, and \(A_{rad}\) is radiative surface area); and convective dissipation follows Newton's law of cooling \(Q_{conv} = h A_{surf} (T - T_m)\) (\(h\) denotes the convective heat transfer coefficient, and \(T_m\) is ambient temperature).
Instead of discretizing continuous space into voxel grids or meshes, ThermoGS analytically computes these thermodynamic geometric terms directly from the semi-major axes \((a, b, c)\) of each 3D Gaussian ellipsoid: - Heat exchange surface area: \(A_{surf} \approx \frac{\pi}{3}(ab + ac + bc)\) - Radiative surface area via Knud Thomsen formula: \(A_{rad} = 4\pi \left(\frac{a^p b^p + a^p c^p + b^p c^p}{3}\right)^{1/p}\) with constant \(p \approx 1.61\) - Solar incident projected area: along the unit solar incidence vector \((d_x, d_y, d_z)\) in the ellipsoid local coordinate system, \(A_{\perp} = \pi \frac{abc}{\sqrt{(a d_x)^2 + (b d_y)^2 + (c d_z)^2}}\) - Primitive volume: \(V = \frac{4}{3}\pi abc\)
By binding thermodynamic terms to explicit Gaussian geometric metrics, the simulated thermal dynamics are rigorously anchored to true metric scales.
2. PINN-Driven Continuous Physical Manifold Inference: Transcending Categorical Lookup Tables
Assigning thermodynamic properties via discrete semantic lookup tables (e.g., standard constants for "building", "road", or "vegetation") completely neglects the pervasive spatial heterogeneity of real-world materials. A single building facade encompasses varying degrees of weathering, surface roughness, structural orientations, and micro-climatic shadowing. ThermoGS employs a deep residual MLP to parameterize a continuous physical property manifold. Receiving Gaussian spatial features, 3D coordinates, high-order Fourier temporal encodings, and synchronized weather vectors as inputs, the network branches into parallel heads predicting the convective coefficient \(h\), emissivity \(e\), volumetric heat capacity \(c\), solar absorptivity \(\alpha\), and absolute temperature \(T\) per Gaussian.
This design enables adaptive refinement of material parameters based on observed spatio-temporal thermal evolutions. Rather than memorizing static snapshot temperatures, the model learns the underlying intrinsic physical manifold, enabling reliable extrapolation to unseen weather conditions and time steps.
3. Multi-Faceted Physics-Informed Joint Constraints: Dual Finite-Difference and ODE Trajectory Alignment
Inverse parameter estimation in transient thermodynamics is fundamentally ill-posed, as multiple parameter combinations can yield similar instantaneous thermal readings. To resolve ambiguities and enforce physical validity, ThermoGS introduces complementary numerical and continuous trajectory constraints: - Finite Difference PINN Loss (\(L_{pinn}\)): Enforces consistency between the numerical temperature rate of change computed across adjacent time intervals and the model's analytical partial derivative \(\frac{\partial T}{\partial t}\) derived via the energy balance PDE. - ODE Integration Consistency Loss (\(L_{ode}\)): Integrates the temperature trajectory over time using a numerical ODE solver, enforcing strict loop-closure alignment with directly predicted temperatures while re-evaluating physical parameters at midpoint intervals. - Physical Prior Regularization (\(L_{physical}\)): Applies soft L1 regularization toward macro-scale empirical reference values for each semantic land-cover class, preventing network optimization from drifting into unphysical local minima.
Loss & Training¶
The overall training loss balances radiometric thermal image reconstruction objectives with physical PDE regularizations: $\(L = \lambda_1 L_{data} + \lambda_2 L_{edge} + \lambda_3 L_{smooth} + \lambda_4 L_{physical} + \lambda_5 L_{ode} + \lambda_6 L_{pinn}\)$ Loss terms and hyperparameter weights are configured as follows: - Data Loss \(L_{data}\) (\(\lambda_1 = 0.8\)): Combination of L1 error and Structural Similarity (SSIM) against radiometric-corrected ground-truth temperature maps. - Edge Loss \(L_{edge}\) (\(\lambda_2 = 0.2\)): Enforces sharp spatial thermal gradients along material boundaries. - Temporal Smoothness Loss \(L_{smooth}\) (\(\lambda_3 = 1.0\)): Suppresses unphysical high-frequency temporal temperature jitter. - Physical Regularization Loss \(L_{physical}\) (\(\lambda_4 = 0.01\)): Soft reference constraint ensuring \(e, h, c, \alpha\) remain within physically plausible ranges. - ODE Consistency Loss \(L_{ode}\) (\(\lambda_5 = 0.01\)): Penalizes drift between continuous ODE trajectories and direct predictions. - PINN PDE Loss \(L_{pinn}\) (\(\lambda_6 = 1.0\)): Directly aligns instantaneous temperature derivatives with thermodynamic energy balance equations.
Key Experimental Results¶
Main Results¶
The experiments are conducted on the newly constructed Thermo Dataset, comprising 40,000 synchronized thermal-RGB frames across four large urban scenes: Substation (S1), School (S2), Square (S3), and Driving School (S4). Table 2 provides quantitative comparisons under cross-time and cross-scene generalization protocols (evaluated with PSNR [dB], SSIM, and Temperature MAE [°C]):
| Scene | Metric | 4DGS (Wu et al. 2024) | NTR-Gaussian (ECCV 2024) | ThermalGS (2024) | Ours (ThermoGS) | Gain (vs. Best Baseline) |
|---|---|---|---|---|---|---|
| S1 (Substation) | PSNR / SSIM / MAE | 24.34 / 0.866 / 2.147 | 25.03 / 0.872 / 1.993 | 24.38 / 0.859 / 2.199 | 26.82 / 0.874 / 1.464 | +1.79 dB / -0.529 °C |
| S2 (School) | PSNR / SSIM / MAE | 29.88 / 0.941 / 1.067 | 27.39 / 0.940 / 1.524 | 30.48 / 0.948 / 1.022 | 31.24 / 0.948 / 0.890 | +0.76 dB / -0.132 °C |
| S3 (Square) | PSNR / SSIM / MAE | 23.66 / 0.949 / 2.533 | 21.78 / 0.944 / 3.122 | 24.22 / 0.952 / 2.333 | 26.03 / 0.955 / 1.490 | +1.81 dB / -0.843 °C |
| S4 (Driving School) | PSNR / SSIM / MAE | 27.37 / 0.934 / 1.403 | 28.80 / 0.950 / 1.278 | 27.42 / 0.944 / 1.426 | 28.81 / 0.954 / 1.173 | +0.01 dB / -0.105 °C |
| S1-S4 Average | PSNR / SSIM / MAE | 28.82 / 0.948 / 1.475 | 31.42 / 0.956 / 1.136 | 29.69 / 0.948 / 1.396 | 31.83 / 0.960 / 0.932 | +0.41 dB / -0.204 °C |
Furthermore, on the original nocturnal NTR Dataset benchmark (Table 3), ThermoGS establishes superior performance across all four regions: S1 achieves 34.62 dB / MAE 0.824°C (vs. NTR-Gaussian 34.50 dB / 0.947°C); S2 reaches 31.85 dB / MAE 0.785°C (vs. 31.43 dB / 1.709°C); S3 reaches 35.41 dB / MAE 0.752°C (vs. 35.15 dB / 0.959°C); and S4 scores 34.94 dB / MAE 0.896°C (vs. 34.61 dB / 1.065°C).
Ablation Study¶
The contribution of each individual loss term was evaluated on the benchmark through systematic component removal (Table 5):
| Configuration | \(L_{edge}\) | \(L_{smooth}\) | \(L_{physical}\) | \(L_{ode}\) | \(L_{pinn}\) | PSNR (dB) ↑ | SSIM ↑ | Temp. MAE (°C) ↓ | Analysis / Finding |
|---|---|---|---|---|---|---|---|---|---|
| w/o \(L_{pinn}\) | ✓ | ✓ | ✓ | ✓ | 31.17 | 0.957 | 1.147 | Removing PDE differential loss breaks heat rate consistency, error climbs to 1.147°C | |
| w/o \(L_{edge}\) | ✓ | ✓ | ✓ | ✓ | 31.27 | 0.957 | 1.163 | Degrades thermal boundary clarity and structural edges | |
| w/o \(L_{smooth}\) | ✓ | ✓ | ✓ | ✓ | 31.33 | 0.956 | 1.147 | Causes high-frequency temporal thermal jitter between adjacent time steps | |
| w/o \(L_{physical}\) | ✓ | ✓ | ✓ | ✓ | 31.42 | 0.956 | 1.175 | Unconstrained physical parameters induce ill-posed optimization, highest error | |
| w/o \(L_{ode}\) | ✓ | ✓ | ✓ | ✓ | 31.42 | 0.956 | 1.133 | Lacks continuous trajectory alignment, leading to temporal drift | |
| Full Model | ✓ | ✓ | ✓ | ✓ | ✓ | 31.83 | 0.960 | 0.932 | Joint integration brings temperature error below 1°C across all metrics |
In the physical attribute prediction ablation (Tables 6 and 7), substituting neural physical inference with rigid categorical table lookup deteriorates temperature MAE from 0.932°C to 1.282°C (PSNR drops to 31.06 dB). Moreover, the predicted physical attributes closely track reference values with remarkable fidelity: emissivity MAE is 0.018, convective transfer coefficient MAE is 0.534 \(\text{W/(m}^2\cdot\text{K)}\), volumetric heat capacity MAE is 5.86 \(\text{kJ/(m}^3\cdot\text{K)}\), and absorptivity MAE is 0.019, vastly outperforming NTR-Gaussian (which exhibits MAEs of 0.470, 32.80, and 160.80, respectively).
Key Findings¶
- Diurnal Temperature Peak Dynamics: Across all four evaluated scenarios, prediction error consistently peaks at 13:00–14:00. This observation aligns directly with thermodynamic principles: peak solar irradiance and maximum ambient temperatures induce rapid, highly nonlinear thermal transients across impervious urban surfaces. By explicitly modeling \(Q_{absorb}\) via projected Gaussian area \(A_\perp\), ThermoGS maintains robust bounds even during these extreme midday periods.
- Computational Feasibility: As detailed in Table 4 and Table 9, ThermoGS requires 20GB of VRAM and achieves 6 FPS per-frame rendering latency on an RTX 4090 GPU, operating within reasonable overhead compared to 4DGS (15GB, 6 FPS) while substantially outperforming NTR-Gaussian in memory efficiency (33GB).
Highlights & Insights¶
- Analytical Geometric-Thermodynamic Equivalence: By deriving analytical expressions for ellipsoid surface area, projected solar incidence cross-section, and primitive volume, ThermoGS directly couples macroscopic energy conservation with the Gaussian Splatting differential pipeline without requiring spatial discretization.
- Transition from Passive Rendering to Interpretable Simulation: Unlike prior radiance-fitting representations, decoupling intrinsic thermal properties yields an interpretable digital twin that can simulate hypothetical futures—such as testing thermal performance under elevated ambient temperatures or modified surface solar reflectance.
- Complementary Dual Physics Constraints: The combination of instantaneous finite-difference matching (\(L_{pinn}\)) and continuous ODE trajectory integration (\(L_{ode}\)) establishes a versatile regularization blueprint applicable to other PDE-governed physical field representations.
Limitations & Future Work¶
- Clear-Sky Environmental Assumption: The current formulation focuses on first-order dominant heat exchanges under clear-sky conditions, omitting complex diffuse scattering induced by variable cloud cover as well as phase change heat transfer (e.g., evaporative cooling from precipitation and condensation).
- Mutual Radiative Exchange and Self-Shadowing: The formulation assumes unobstructed solar vectors per primitive without computing ray-traced geometric occlusion (cast shadows from adjacent high-rises) or mutual infrared reflections between opposing surfaces.
- Future Directions: Integrating lightweight neural shadow mapping to simulate dynamic solar occlusion and extending the energy balance equations to include latent heat transfer for wet or vegetated surfaces represent key avenues for future enhancement.
Related Work & Insights¶
- vs. NTR-Gaussian (ECCV 2024): NTR-Gaussian introduced dynamic thermal reconstruction restricted to nighttime conditions. Because it ignores solar radiation, its daytime error degrades severely (MAE reaches 3.12°C on S3); ThermoGS incorporates solar absorption and Gaussian projected areas, enabling accurate full-day simulation.
- vs. Thermal3D-GS / ThermalGS: These approaches extend 3DGS to thermal novel-view synthesis by regressing radiance as static view-dependent color. ThermoGS generalizes thermal modeling into a 4D spatio-temporal physics-informed simulation framework with decoupled intrinsic material properties.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First framework to couple micro-scale 3D Gaussian ellipsoid geometry with thermodynamic energy balance equations, successfully decoupling physical surface attributes.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on 40K real-world drone multimodal thermal frames across 4 large urban environments, rigorously assessing cross-time, cross-scene, and physical property inference.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous thermodynamic formulations, clear mathematical grounding, and well-structured experimental validations.
- Value: ⭐⭐⭐⭐⭐ Holds immediate practical impact for urban digital twins, building energy auditing, UAV surveillance, and autonomous driving simulation in all-weather scenarios.