Gaussians on Fire: High-Frequency Reconstruction of Flames¶
Conference: ECCV 2026
Paper: ECCV
Area: 3D Vision
Keywords: Gaussian Splatting, flame reconstruction, sparse-view reconstruction, 4D dynamic reconstruction, transient volumetric phenomena
TL;DR¶
The paper redefines fire as a population of short-lived volumetric emission carriers rather than persistent scene primitives: each 3D Gaussian carries a lifetime and a linear velocity, is initialized from a 3D flow field fused from three-view dense optical flow, and is captured with an ESP32 + LED sub-frame synchronization rig, so that three consumer cameras suffice to reconstruct a temporally coherent high-frequency 4D flame representation that beats 4DGS, Grid4D and FreeTimeGS on flame-region PSNR/SSIM/LPIPS, depth consistency, and motion-field similarity on synthetic scenes.
Background & Motivation¶
Differentiable scene representations have pushed static 3D reconstruction to a remarkably mature level: 3D Gaussian Splatting renders view-dependent anisotropic Gaussians through a differentiable splatting pipeline and delivers high-quality novel-view synthesis at real-time rates, and has become the de facto standard. Following this line, dynamic scenes were extended into 4D by a large body of work — 4DGS learns 4D Gaussian primitives, FreeTimeGS lets primitives appear at arbitrary times, Grid4D trades in decomposed hash encodings for higher dynamic fidelity, and articulated objects such as humans and animals have mature deformation-field solutions. All of these methods share one implicit assumption: scene content can be written as a set of persistent primitives undergoing smooth, trackable motion.
Fire violates that assumption on every count. It has no stable surface correspondence across frames — this frame's flame and the "same" flame in the next frame are not the same physical entity; it is semi-transparent volumetric emission rather than a surface; it is self-emissive (brightness depends on combustion and temperature and cannot be explained by a surface reflectance model); and it continuously changes shape and topology at high frequency. Comparing against 4DGS (Wu et al.), 4DGS (Yang et al.), Grid4D and FreeTimeGS, the authors observe three recurring failure modes: geometric absorption (flame motion is explained by deforming the background, so the fire is "absorbed" into the static scene), depth collapse (the rendered images look acceptable while the underlying 3D structure is meaningless), and temporal drift (the rendered flame evolution no longer matches the captured observation). The capture side is just as awkward: real fire cannot easily be surrounded by many cameras, consumer cameras expose no hardware sync interface, and at high frame rates the inter-frame displacement of flames is large enough that any timing offset turns directly into false geometric supervision. The paper therefore tightens the problem to an extreme setting — three calibrated and synchronized cameras, sparse views, and severely under-constrained geometry.
The angle is not "swap in a stronger renderer or bolt on a physics simulator" — the authors explicitly stress that the design is deliberately minimal. What is actually replaced is the unit of representation. Since flame kinematics is not "a surface moving," persistent primitives should not be used to fit it. The scene is split into two layers: the static background is first pinned down by dense geometry fused from stereo matching and monocular depth priors, and the dynamic part is represented as transient flame Gaussians — short-lived volumetric emission carriers that exist only over a finite temporal window, whose positions are driven by an explicit linear velocity that is not learned from scratch but initialized from a 3D flow field obtained by fusing back-projected dense optical flow from three views. Core idea: rewrite fire from "persistent primitives being deformed" into "transient volumetric Gaussians with a lifetime and a linear velocity, initialized from multi-view dense motion evidence," so that the motion prior which photometric supervision cannot learn under sparse views is injected at initialization time.
Method¶
Overall Architecture¶
The input is video of a fire captured by three calibrated, synchronized cameras; the output is a coupled 4D representation consisting of a static-background 3DGS plus a time-varying field of transient flame Gaussians. This supports photo-realistic novel-view synthesis, re-rendering at arbitrary time steps, and — because each Gaussian's velocity is an explicit parameter — direct reading of motion trajectories for downstream analysis (e.g. robot navigation that has to avoid a fire source).
The pipeline has four parts. On the capture side, a custom ESP32 + LED rig solves inter-camera frame synchronization. On the reconstruction side, the static background and the dynamic flame are decoupled: the background is initialized from a dense point cloud fused from stereo and monocular depth and then optimized with vanilla 3DGS. The motion prior for the dynamic region comes from back-projection fusion of three dense optical flows, yielding a voxel-level 3D flow field. Finally this flow field initializes the transient flame Gaussians (position from the voxel, velocity from the flow vector, lifetime of a few frames), which are jointly optimized with the background under photometric and mask supervision. These four parts correspond exactly to the four key designs below.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Three-view fire video<br/>400 Hz consumer cameras"] --> B["Sub-frame visual sync<br/>ESP32 + LED timing code"]
B --> C["Static / dynamic decoupling<br/>stereo + monocular depth fusion"]
C --> D["Static background 3DGS"]
C --> E["3D flow field initialization<br/>multi-view flow back-projection"]
E --> F["Transient flame Gaussians<br/>lifetime + linear velocity"]
D --> G["4D flame reconstruction<br/>novel views / velocity analysis"]
F --> G
Key Designs¶
1. Sub-frame visual sync: aligning consumer cameras with no sync interface to microsecond precision
At 400 Hz the inter-frame displacement of a flame is enormous; if two cameras are even half a frame apart, the same flame volume lands at different positions in different views, and multi-view supervision degenerates into "mutually correcting misaligned data," which destroys the geometry. Consumer cameras such as the GoPro have no frame-sync interface, so the authors build the hardware themselves: an ESP32 microcontroller drives 21 logic-level pins, five of which produce a sub-frame synchronization pattern by sequentially toggling five COB LED strips, while the remaining 16 maintain an incrementing frame counter — 15 white LEDs display the frame number in Gray code, and the 16th LED exists solely to resolve the inherent ambiguity of Gray code at counting transitions (without it the decoded frame index can be off by one).
Sub-frame precision is bought from the rolling shutter: a CMOS sensor exposes row by row, so a toggling LED strip appears partially illuminated, and the row at which the brightness changes directly encodes the exposure instant. With a row time of about 3 μs and a localization accuracy of about 5 pixels, the resulting temporal precision is roughly 15 μs — two to three orders of magnitude below the frame interval. The pattern is masked out during both training and evaluation, so it never contaminates supervision. The ablation that shifts timing by half a frame (NoSync) degrades flame-region visual quality on real scenes and motion-field accuracy on synthetic scenes (CosSim 0.736 vs 0.744), confirming this is a prerequisite rather than a nicety.
2. Static / dynamic decoupling: pinning down the background with fused depth first
Three cameras already leave the background under-constrained; if flames and background share one set of movable primitives, the optimizer's easiest shortcut is to explain flame motion as background deformation — exactly the "geometric absorption" described above. The countermeasure is to fix the static part first. Frames without motion are used when available; otherwise the method exploits the physical fact that flames are self-emissive and brighter than their background, and estimates a clean background with a minimum intensity projection along the time dimension. Poses come from COLMAP, dense geometry from COLMAP's PatchMatchStereo implementation, and monocular depth from DepthAnythingV2, which is then affinely aligned per camera:
where \(a,b\) are fitted by least squares over regions where stereo and monocular depth overlap (Eq. 1 in the paper; that equation is garbled in the cached text, ⚠️ refer to the original paper). Back-projecting the three stereo depth maps yields the initial point cloud, and regions without stereo information are filled with aligned monocular depth; the background is optimized with vanilla 3DGS, with the aligned monocular depth added as a regularizer during training,
to prevent degenerate depth solutions under three views. Intuitively, a dense trustworthy geometric initialization plus a persistent depth constraint writes "the background must not move" into the objective, and only the remaining degrees of freedom are left to the flames.
3. 3D flow field initialization: back-projecting three dense optical flows into a voxel-level motion field
Dense stereo cannot solve the dynamic part for two reasons: the camera frames are not synchronized, and stereo matching does not produce a continuous motion field. The authors therefore estimate a rough motion field first and let the joint optimization refine it. Concretely, dense optical flow \(\mathbf{f}_i(u,v)\) (a per-pixel 2D displacement) is computed between consecutive frames of each camera with MEMFOF, and all 2D flows are projected onto a voxel grid: for each 3D point, back-projecting it into the image plane of each camera at the appropriate depth yields one piece of 3D motion evidence per view. With only three views these constraints are neither sufficient nor mutually consistent, so the system is written as a linear system with Tikhonov regularization to suppress unstable solutions under under-constrained conditions, with the regularization strength tied to the mean flow residual across cameras:
(Eqs. 5–6 in the paper; the cached equations are garbled, ⚠️ refer to the original paper); the final 3D flow is obtained by solving this least-squares system via singular value decomposition. The ablation provides blunt evidence for why this step is indispensable: if the dynamic Gaussians' velocities are initialized randomly (RandFlow), photometric supervision alone cannot recover the true velocity field — on synthetic scenes CosSim drops from 0.744 to 0.338 and L2 rises from 0.825 to 1.519. In other words, dense optical flow is reliable in image space and directly encodes the displacement of flame texture, so the cross-view fused motion field is a physically sensible prior instead of something the optimizer has to guess from pixel colors.
4. Transient flame Gaussians: carrying high-frequency emission with a lifetime and a linear velocity
This is where the paper's representation lands. Since flames have no persistent surface correspondence across frames, treating fire as "persistent primitives being deformed" inevitably drifts temporally. Each Gaussian is given three extra parameters: a time \(t_\mu\), a lifespan \(t_\sigma\), and a linear velocity \(\mathbf{v}\). Its position at any time is extrapolated at constant velocity,
and its opacity is further modulated by a temporal term that decays with the distance from \(t_\mu\), \(\sigma(t)=\exp(\cdot)\) (the exact form is garbled in the cached text, ⚠️ refer to the original paper). Each primitive therefore "lives" only inside a finite temporal window — these are the transient flame Gaussians. The representation has two immediate benefits: its parameters can be initialized directly from the 3D flow field (position from the voxel, velocity from the flow vector, \(t_\mu\) from the current time), injecting the velocity prior that sparse views cannot learn; and the velocity field is explicit, so it supports downstream tasks such as trajectory analysis. Two initialization details matter: the lifespan is set to four times the frame interval, which keeps motion smooth while preventing overfitting to a single frame; and the position gets a uniform random offset inside the voxel volume, \(\Delta\mathbf{x}\sim\mathrm{Uniform}([-s/2,s/2]^3)\) with \(s\) the voxel size, to remove moiré patterns caused by the regular grid.
To further keep dynamic Gaussians from growing into static regions, training compares the accumulated opacity of dynamic Gaussians \(A_{\text{dyn}}\) against a precomputed motion mask \(M\) and adds an alpha loss with weight 0.1, penalizing dynamic Gaussians placed where the input video contains no flame at all. The ablation exposes an instructive trade-off: removing this loss (NoMask) actually improves the visual metrics (flame PSNR 27.34 vs 26.93) while clearly degrading the motion field (CosSim 0.697 vs 0.744) — without the mask the model can quietly place dynamic Gaussians in static regions to "subsidize" image quality, so this loss is effectively a trade of visual fidelity for physical correctness.
Loss & Training¶
The static scene keeps the 3DGS objective with an added depth regularizer, giving a weighted sum of three terms:
with \(\lambda_1=0.8\) and \(\lambda_{\text{SSIM}}=0.2\). To genuinely constrain geometry under three views, the depth regularization weight is increased by a factor of 100 over the 3DGS default and decays exponentially from an initial value of 100 to a final value of 1 (matching the default 3DGS regularization schedule). The dynamic part adds the aforementioned alpha loss with weight 0.1 (dynamic Gaussian opacity against the motion mask) on top of the photometric loss. Dynamic Gaussians are initialized entirely from the 3D flow field: position from the corresponding voxel (plus the random offset), velocity from the fused flow vector at that voxel, timestamp from the current time, and lifespan of four frame intervals. For evaluation, every 8th frame is held out for validation. Rasterization uses the original 3DGS linear transmittance approximation rather than exponential attenuation; the authors justify this: after flow-based initialization the dynamic scene consists of a large number of small Gaussians (roughly 300 per ray), for which the linear approximation is very close to the exponential model, so the added complexity of an exponential rasterizer is not justified.
Key Experimental Results¶
The data comes in two parts. The real dataset is self-captured: three GoPro Hero 13 Black cameras in a fixed multi-view rig, each recording at 400 Hz, with sequences of roughly 15 s covering propane jets, burning wood, gasoline, cardboard and paper against different backgrounds — 17 fire videos per camera, with a 100-frame subsequence taken from each scene. The synthetic data serves motion-field evaluation: three fire scenes simulated in Mantaflow, placed in three Blender scenes with three cameras each, rendering 50 ray-traced frames per camera (2 s of simulation) and providing a ground-truth velocity field at resolution \(640^3\), for 150 synthetic velocity fields in total. Baselines are 4DGS (Wu et al.), 4DGS (Yang et al.), Grid4D and FreeTimeGS; all are initialized from COLMAP points except FreeTimeGS, which uses its own RoMa initialization, and hyper-parameters follow the authors' configuration for the DyNeRF flame salmon scene.
Beyond standard PSNR/SSIM/LPIPS, two groups of metrics are tailored to this problem. The first are flame-region masked metrics, PSNR_flame and SSIM_flame, which only evaluate regions of the image that contain motion so that a large static background cannot dilute the differences (the synchronization pattern is masked out when computing PSNR/SSIM). The second is depth consistency, RMSE_depth, comparing rendered depth against a linearly aligned monocular depth prediction. Moreover, because image metrics cannot directly measure whether a dynamic 3D scene moves correctly, the authors propose the density-weighted volumetric evaluation protocol (DVE): each Gaussian gets a weight \(w=s_x s_y s_z \alpha\) from its scale and opacity, the ground-truth velocity field's expectation under that Gaussian's probability distribution \(\bar{\mathbf{v}}=\mathbb{E}_P(V)\) is approximated with third-order Gauss–Hermite quadrature, and magnitude error and directional agreement are measured jointly:
(the exact normalization is garbled in the cached text, ⚠️ refer to the original paper). The point of this protocol is that frame-wise image metrics alone are insufficient: some methods look acceptable in PSNR while their 3D structure and temporal behavior have already collapsed.
Main Results¶
Comparison on the real-world dataset (every 8th frame held out for validation):
| Method | PSNR_flame ↑ | SSIM_flame ↑ | RMSE_depth ↓ | LPIPS ↓ | PSNR ↑ | SSIM ↑ |
|---|---|---|---|---|---|---|
| Ours | 26.93±1.93 | 0.843±0.048 | 0.094±0.038 | 0.035±0.011 | 35.89±2.62 | 0.967±0.011 |
| 4DGS (Yang et al.) | 23.28±1.76 | 0.728±0.069 | 0.113±0.045 | 0.053±0.020 | 32.86 | — |
| 4DGS (Wu et al.) | 20.99±1.14 | 0.601±0.054 | 0.036 | 0.071±0.025 | 31.88±3.12 | 0.933±0.025 |
| FreeTimeGS | 23.48 | 0.132 | 0.131±0.042 | 0.241±0.131 | 25.62±1.83 | 0.794±0.109 |
| Grid4D | 18.52±1.47 | 0.497±0.068 | 0.225 | 0.086±0.025 | 29.16 | 0.909±0.032 |
⚠️ Several cells above are mis-parsed from the cached PDF text (marked "—"); refer to Tab. 1 of the original paper for the values. The confirmable qualitative conclusion is that Ours is best on flame-region metrics, full-frame metrics and depth consistency.
DVE motion-field evaluation on the synthetic scenes (ground-truth velocity fields from Mantaflow):
| Method | L2 ↓ | CosSim ↑ |
|---|---|---|
| Ours | 0.825±0.058 | 0.744±0.032 |
| 4DGS (Yang et al.) | 1.045±0.194 | 0.225±0.123 |
| FreeTimeGS | 1.164±0.042 | 0.194±0.114 |
| 4DGS (Wu et al.) | 2.218±0.276 | 0.036±0.010 |
| Grid4D | 2.262±1.470 | 0.154±0.014 |
Ablation Study¶
Ablations on the real data (lower half of Tab. 1) alongside the synthetic motion metrics:
| Config | PSNR_flame ↑ | LPIPS ↓ | PSNR ↑ | CosSim ↑ | Note |
|---|---|---|---|---|---|
| Full model | 26.93±1.93 | 0.035±0.011 | 35.89±2.62 | 0.744±0.032 | full model |
| NoSync | 26.89±2.00 | 0.035±0.011 | 35.91±2.67 | 0.736±0.034 | half-frame timing offset; both flame-region visuals and motion accuracy drop |
| RandFlow | 26.68±2.43 | 0.036±0.013 | 35.67±3.21 | 0.338±0.050 | random velocity initialization; the motion field essentially collapses |
| JointOpt | 26.52±1.89 | 0.060±0.014 | 33.59±1.89 | 0.749±0.033 | no static/dynamic separation, joint optimization; the largest full-frame drop |
| NoMask | 27.34±1.98 | 0.034±0.011 | 36.28±2.72 | 0.697±0.035 | without the alpha mask loss; better visuals, worse motion field |
Key Findings¶
- Flow initialization is what makes the motion correct: the RandFlow ablation is the most telling — without flow initialization the real-scene visual metrics barely move (flame PSNR 26.68 vs 26.93), yet the synthetic directional agreement collapses from 0.744 to 0.338 and L2 rises from 0.825 to 1.519. Under sparse views, photometric supervision alone cannot recover the velocities; the prior has to come from multi-view dense optical flow.
- Visual fidelity and physical correctness genuinely trade off: NoMask attains the best visual scores (flame PSNR 27.34, full-frame PSNR 36.28) but its motion CosSim falls from 0.744 to 0.697. The alpha mask removes the model's freedom to place dynamic Gaussians in static regions to inflate image quality, so this loss buys physical correctness with image quality rather than being a free gain.
- Decoupling mainly buys full-frame quality: JointOpt's degradation is concentrated in full-frame metrics (PSNR 33.59 vs 35.89, LPIPS 0.060 vs 0.035), while its synthetic motion metrics are in fact slightly better (CosSim 0.749). Static pretraining therefore contributes mostly to background/full-image fidelity rather than motion accuracy — a point the authors state explicitly.
- Synchronization is necessary, not optional: NoSync loses quality mainly in the flame region on real data, and its synthetic degradation is concentrated in the motion metrics (CosSim 0.736), consistent with the claim that timing offsets turn directly into false geometric supervision.
- Baselines produce "plausible images with broken structure": FreeTimeGS achieves second-best flame reconstruction (PSNR_flame 23.48) at the cost of severe depth degradation (RMSE_depth 0.131), and 4DGS (Yang et al.) ranks second on full-frame metrics. All baselines show weak directional agreement (best CosSim 0.225), indicating they reconstruct images that look like fire rather than the motion of fire.
Highlights & Insights¶
- Changing the unit of representation instead of the renderer: the paper's most valuable move is attributing dynamic-Gaussian failure on flames to the representation's implicit assumption (persistent primitives plus smooth motion) and then changing only that unit — adding a timestamp, a lifespan and a linear velocity while keeping vanilla 3DGS rasterization throughout. Such a minimal, precisely aimed change is far easier to reproduce and transfer than bolting on a physics simulator.
- Initialization as a channel for injecting priors: velocity would normally be a parameter learned by optimization, but here it is assigned directly from a 3D flow field fused from back-projected dense optical flow, with engineering details such as a four-frame lifespan and a random within-voxel offset. The RandFlow ablation proves the design is necessary, and the general idea — fixing motion priors from geometric evidence rather than from gradients — transfers to smoke, fluids, and any transient volumetric phenomenon.
- Buying temporal precision with cheap hardware: ESP32 plus Gray-code LED frame counting plus rolling-shutter readout of the exposure instant turns consumer cameras into a microsecond-synchronized capture array. The whole rig is inexpensive yet solves the messiest part of sparse multi-view dynamic reconstruction, and the pattern is masked out so it never affects any metric — a clean reusable engineering trick.
- A custom protocol that targets the blind spot of image metrics: DVE weights by scale times opacity and references the ground-truth velocity expectation under each Gaussian distribution, reporting both magnitude error and directional agreement. Because a Gaussian is a spatial distribution rather than a point mass, tracking center positions alone does not capture its dynamics — an observation that holds for all work representing fluids or smoke with Gaussians.
Limitations & Future Work¶
- Heavy dependence on dense optical flow: where emissive variation is strong or texture is weak, the flow itself is inaccurate and the error propagates into the 3D motion field. Worse, flames missed by the flow estimator during initialization cannot be recovered by the later dynamic optimization — a single point of failure for the whole pipeline.
- The linear velocity model is too simple: real flames are turbulent, so constant-velocity extrapolation leads to temporal blurring or oversmoothing. The authors propose integrating physics-based priors for combustion dynamics and more adaptive temporal parameterizations, including learning-based flow estimation directly in 3D space.
- Linear transmittance approximation: the pipeline still uses 3DGS's linear transmittance instead of exponential attenuation. The justification — roughly 300 small Gaussians per ray after initialization, making the two numerically close — is empirical, and it is not verified for thinner or thicker optical conditions such as smoke.
- The static/dynamic separation assumes weak foreground-background interaction: this likely fails for large-scale or smoke-rich scenes (background occluded by smoke, illuminated by fire), which would break the premise of decoupling.
- Residual synchronization error: the LED scheme removes frame-level to sub-frame offsets, but sub-millisecond uncertainty can still introduce artifacts at very high motion speeds; the authors list joint optimization of per-camera timing offsets as future work.
- From a reading standpoint, one more caveat: the synthetic evaluation covers only three Mantaflow scenes and 150 velocity fields whose ground truth is simulated rather than measured, so DVE absolute values (e.g. CosSim 0.744) should not be read as a general level for real flames.
Related Work & Insights¶
- vs FreeTimeGS: it is the most direct source of this paper's ideas — the same time, lifespan and linear velocity attached to each Gaussian. The difference is that FreeTimeGS targets general dynamic scenes, where primitives are still treated as elements that persist and can be tracked through smooth motion and are initialized by RoMa matching, whereas this paper restricts primitives to short-lived transient carriers and replaces initialization with a 3D flow field fused from multi-view dense optical flow. The cost is that this method only applies to transient volumetric phenomena under sparse views and has no advantage on rigid or articulated dynamic scenes.
- vs 4DGS (Wu et al. / Yang et al.) and Grid4D: they model dynamics through 4D primitives or decomposed space-time representations, which are still persistent representations that deform or interpolate; on flames this surfaces as geometric absorption, depth collapse and temporal drift. This paper instead hands the dynamic part entirely to transient primitives with an explicit existence window. The downside is that it needs extra static/dynamic separation and masking, making the pipeline longer and more hyper-parameter-heavy.
- vs physics-informed neural fields and tomographic flame reconstruction (e.g. Chu et al. on smoke, Ihrke & Magnor on flames): those methods are far stronger in physical consistency but rely on controlled multi-view setups or simplified motion models. This paper goes the other way and targets three consumer cameras, trading physical rigor for capture feasibility.
- vs free-viewpoint rendering of dynamic scenes from unsynchronized inputs with a temporal Laplacian pyramid: such methods render convincing free-viewpoint fire but do not target a full spatiotemporal reconstruction of geometry and appearance and cannot expose an explicit velocity field. Here the velocity field is an explicit parameter, which is the key difference from pure rendering approaches.
Rating¶
- Novelty: ⭐⭐⭐⭐ The representational change is modest (time, lifespan and velocity are borrowed from FreeTimeGS), but the choice of transient volumetric carriers is woven together with the physics of fire, sparse-view capture, a sub-frame synchronization rig and an evaluation protocol into a coherent argument — the problem definition outshines the method itself.
- Experimental Thoroughness: ⭐⭐⭐⭐ A self-captured set of 17 real fire videos × 3 views plus Mantaflow synthetic velocity fields, four metric families (image / flame region / depth / motion field) and four ablations give broad coverage; however only three synthetic scenes with simulated ground truth, plus the partially garbled cached table, weaken reproducibility checks.
- Writing Quality: ⭐⭐⭐⭐ The three failure modes, the ablation conclusions and the honest discussion of the visual-physical trade-off are all strong; some equations are dense as typeset, and many details (scene decomposition, exponential rasterizer comparison, failure modes) are pushed to the supplementary material.
- Value: ⭐⭐⭐⭐ It offers a very low-cost, reproducible reconstruction paradigm for phenomena that are semi-transparent, self-emissive, high-frequency and lack persistent correspondence; the capture-side synchronization scheme and the DVE protocol transfer directly to smoke, steam and fluids.