HybridSim: A Physics–Learning Hybrid Digital Twin for mmWave Human Sensing

Conference: ECCV 2026
Paper: ECCV Official
Code: https://weitao-xiong.github.io/HybridSim/
Area: 3D Vision
Keywords: mmWave Radar, Human Sensing, Physics-Learning Hybrid Simulation, Microfacet BRDF, 3D Gaussian Splatting

TL;DR

Addressing the exponential ray-tracing overhead and neural-implicit motion entanglement in radar simulation, HybridSim decouples electromagnetic wave propagation into an inverse-rendering direct path with microfacet BRDF and a 3D Gaussian Splatting indirect path with proxy virtual receivers, synthesizing high-fidelity intermediate-frequency mmWave signals at under 1 second per frame.

Background & Motivation

Millimeter-wave (mmWave) radar offers a compelling alternative to optical and vision-based sensing systems in human-centric applications, providing remarkable resilience to adverse illumination, smoke, and inclement weather while inherently preserving personal privacy. However, training robust deep neural networks for radar-based human sensing demands massive datasets covering diverse body geometries and complex articulated motions. Collecting and accurately annotating such data in physical environments remains prohibitively expensive and labor-intensive. Generating synthetic radar data through digital twins presents a scalable solution to enrich training distributions and bridge the sim-to-real domain gap.

Existing simulation paradigms face severe fundamental limitations when synthesizing signals for continuous, articulated human motion. On one hand, classical physics-based geometric ray tracing accurately models electromagnetic wave propagation, but its computational burden explodes exponentially when calculating higher-order multipath interactions between the human body and room boundaries; furthermore, it demands meticulously scanned scene meshes that are impractical for ubiquitous deployment. On the other hand, recent neural radio-frequency representations—adapting neural radiance fields (NeRF2) or 3D Gaussian Splatting (RF-3DGS)—excel at static site-specific channel modeling but inherently entangle the primary surface reflections of dynamic bodies with complex background multipath reflections. In single-sensor configurations, this entanglement severely degrades the time-varying micro-Doppler signatures that are vital for distinguishing articulated limb kinematics.

The key insight of this paper is that mmWave wave-body interactions can be naturally factorized into two complementary regimes: line-of-sight surface scattering that governs high-frequency kinematic micro-Doppler signatures, and indirect environmental multipaths that govern site-specific coherent interference textures. Core idea: decouple mmWave propagation into an inverse-rendering direct path constrained by microfacet BRDF and an indirect multipath surrogate using geometrically flattened 3D Gaussian Splatting paired with boundary proxy virtual receivers, optimized end-to-end on decibel-scaled Range-Doppler heatmaps.

Method

Overall Architecture

HybridSim takes dynamic human SMPL mesh sequences and coarse room boundary configurations (scalar transceiver-to-wall distances) as inputs, synthesizing high-fidelity intermediate-frequency (IF) complex radar signals and Range-Doppler (RD) heatmaps. As illustrated in the pipeline, the framework first extracts canonical pose features using a tri-plane representation and resolves dynamic self-occlusion via Hidden Point Removal (HPR). Next, propagation is bifurcated into two parallel, decoupled branches: the direct path employs Graph Convolutional Network (GCN) feature regularization and a microfacet BRDF decoder to estimate single-bounce surface reflections; concurrently, the indirect path combines surface-aligned 2D Gaussian patches with proxy virtual receivers placed on boundary planes to formulate a differentiable, three-segment propagation surrogate. Finally, direct and indirect waveforms are coherently superimposed with a learnable complex Gaussian noise floor and supervised in the logarithmic RD domain.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inputs: Dynamic SMPL Meshes + Room Scalar Bounds"] --> B["Triplane Feature Extraction & HPR Dynamic Visibility"]
    B --> C["Direct Path: Regularized Microfacet Inverse Rendering<br/>Mesh GCN Smoothing + Microfacet BRDF Decoding"]
    B --> D["Indirect Path: 3DGS Multi-Scatterer Surrogate with Proxy Receivers<br/>Surface-flattened 2D Gaussians + Boundary Virtual Receivers"]
    C --> E["Coherent Signal Superposition & Complex Gaussian Noise Floor"]
    D --> E
    E --> F["Amplitude-Domain Logarithmic RD Heatmap Supervision<br/>2D-FFT Transform + 20 log10(·) Decibel Scale MSE"]
    F --> G["Outputs: High-Fidelity IF mmWave Signals & RD Heatmaps"]

Key Designs

1. Regularized Microfacet Inverse Rendering for Direct Path: Isolating Fine-Grained Micro-Doppler Kinematics

The direct signal path strictly models single-bounce line-of-sight (1-bounce LoS) reflections from the moving body and room boundaries back to the radar receiver. To ensure pose-invariant feature extraction, canonical SMPL vertices are mapped into orthogonal tri-plane feature spaces. However, microfacet BRDF parameters—such as surface roughness, metallic factor, and complex refractive indices—are notoriously sensitive to gradient updates during inverse rendering. Treating mesh vertices independently causes severe gradient concentration on high-velocity limb vertices, provoking catastrophic optimization instability and memory spikes.

To resolve this, HybridSim applies a Graph Convolutional Network (GCN) over the mesh graph with stochastic feature masking during training, enforcing a Laplacian-like spatial smoothness prior across the human surface. Crucially, this GCN operates strictly as a training-time regularizer: during inference, latent features are decoded directly without graph propagation, preventing unwanted spatial over-smoothing that would blur legitimate high-frequency scattering shifts. For each visible scatterer \(p \in \mathcal{P}_{\text{dir}}\), the scattering amplitude \(L_p\) is computed via microfacet BRDF formulations, while the phase is physically dictated by the propagation delay \(\tau_p(t)\):

\[\tau_p(t) = \frac{\|\mathbf{x}_{\text{tx}} - \mathbf{x}_p(t)\| + \|\mathbf{x}_p(t) - \mathbf{x}_{\text{rx}}\|}{c}\]

The aggregate direct complex signal is formulated as the coherent sum across all visible scatterers:

\[\text{Sim}_d(t) = \sum_{p \in \text{vis}} \frac{L_p \cdot \exp\left(j 2\pi \left(f_0 \tau_p(t) + \frac{1}{2} S t \tau_p(t)\right)\right)}{\|\mathbf{x}_{\text{tx}} - \mathbf{x}_p(t)\| \cdot \|\mathbf{x}_p(t) - \mathbf{x}_{\text{rx}}\|}\]

where \(f_0\) is the radar carrier frequency, \(S\) is the chirp slope, and the denominator explicitly models inverse-distance free-space attenuation for a single-pixel radar point sink.

2. 3DGS Multi-Scatterer Surrogate with Proxy Receivers for Indirect Path: Taming Multi-Bounce Combinatorial Explosion

The indirect path is dedicated exclusively to higher-order multipath reflections (\(\ge 2\) bounces), including body-to-wall, floor-to-wall, and inter-wall interactions. Tracking these paths with explicit ray tracing incurs exponential computational overhead and introduces non-differentiable discrete ray discontinuities. Unconstrained 3D Gaussian Splatting also fails because free Gaussians easily inflate into non-physical volumetric clouds.

HybridSim introduces a unified multi-scatterer formulation paired with proxy boundary geometry. The dynamic body mesh and static room boundary point clouds are discretized into a unified scatterer set \(\mathcal{P}_{\text{ind}} = \mathcal{P}_{\text{human}} \cup \mathcal{P}_{\text{env}}\). All Gaussian primitives are flattened into 2D surface patches aligned with local tangent planes and compressed along surface normals to preserve physical topology. To model indirect paths without ray tracing, proxy virtual receivers \(\mathbf{x}_{w,k}\) are instantiated on each boundary surface \(k \in \mathcal{K}\). This simplifies the intractable multi-bounce integral into a differentiable three-segment propagation link: \(\mathbf{x}_{\text{tx}} \to \mathbf{x}_p(t) \to \mathbf{x}_{w,k} \to \mathbf{x}_{\text{rx}}\). The 3DGS decoder predicts directional spherical harmonics \(SH_{p,k}\) toward each proxy receiver, and the global indirect signal is integrated as:

\[\text{Sim}_{\text{ind}}(t) = \sum_{k \in \mathcal{K}} \sum_{p \in \text{vis}(k)} \frac{SH_{p,k} \cdot \exp\left(j 2\pi \left(f_0 \tau_{p,k}(t) + \frac{1}{2} S t \tau_{p,k}(t)\right)\right)}{\|\mathbf{x}_{\text{tx}} - \mathbf{x}_p(t)\| \cdot \|\mathbf{x}_p(t) - \mathbf{x}_{w,k}\| \cdot \|\mathbf{x}_{w,k} - \mathbf{x}_{\text{rx}}\|}\]

with proxy delay \(\tau_{p,k}(t) = (\|\mathbf{x}_{\text{tx}} - \mathbf{x}_p(t)\| + \|\mathbf{x}_p(t) - \mathbf{x}_{w,k}\| + \|\mathbf{x}_{w,k} - \mathbf{x}_{\text{rx}}\|)/c\). Here, geometric kinematics are captured by \(d\tau/dt\), while unmodeled electromagnetic attenuation and site-specific multipath textures are absorbed into the neural spherical harmonics.

3. Amplitude-Domain Decibel-Scaled RD Supervision with Learnable Noise Floor: Overcoming Phase Misalignment and Dynamic Range Imbalance

Supervising raw time-domain complex signals via Mean Squared Error (MSE) fails completely in mmWave regimes: at 77 GHz (\(\lambda \approx 3.9\text{ mm}\)), sub-millimeter mesh tracking errors distort the absolute phase by up to \(180^\circ\), rendering gradients chaotic. In addition, radar signals exhibit massive dynamic range, causing strong primary reflections to overpower faint limb micro-Doppler signals.

HybridSim addresses this via a threefold strategy: first, it abandons absolute phase matching by applying a 2D Fast Fourier Transform (2D-FFT) across range and Doppler dimensions, supervising only the resulting spectral amplitude \(|\mathcal{F}_{\text{RD}}|\); second, it injects a zero-mean complex Gaussian noise floor \(\mathcal{CN}(0, \sigma^2)\) into the idealized time-domain signal, where \(\sigma\) is dynamically scaled by a learnable rate against the mean signal amplitude; third, it applies a \(20 \log_{10}(\cdot)\) decibel transformation to compress the dynamic range, allowing gradient descent to balance prominent specular peaks and subtle micro-Doppler signatures.

Loss & Training

The framework is optimized end-to-end using Mean Squared Error on decibel-scaled Range-Doppler heatmaps:

\[\mathcal{L} = \frac{1}{N} \sum_{n=1}^{N} \left| 20 \log_{10}\left(\left|\mathcal{F}_{\text{RD}}(\text{Sim})\right|_n\right) - 20 \log_{10}\left(\left|\mathcal{F}_{\text{RD}}(\text{GT})\right|_n\right) \right|^2\]

where \(N\) is the total number of bins in the flattened RD matrix. The entire framework is trained on a single NVIDIA RTX 4090 GPU (peak memory \(\approx 20\text{ GB}\)) in roughly 7 hours. At test time, synthesis requires less than 1 second per frame.

Key Experimental Results

Main Results

Evaluations are conducted on the public mmMesh dataset (collected via a TI AWR1843BOOST 3TX-4RX radar) covering 8 distinct action categories. HybridSim is benchmarked against physical simulation baseline mmGPE and diffusion-augmented ray tracer RF-Genesis. Metrics include RD heatmap reconstruction quality (PSNR, SSIM, LPIPS) and downstream Human Activity Recognition (HAR) accuracy evaluated on real-world measurements using an mmAP classification backbone trained purely on synthetic data.

Setting	Method	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	HAR Acc (%) ↑	HAR F1-Score ↑
Intra-Subject (Unseen Frames)	RF-Genesis	20.03	0.219	0.274	—	—
	mmGPE	20.09	0.221	0.261	—	—
	HybridSim (Ours)	22.30	0.262	0.089	—	—
Cross-Subject (Unseen Human)	RF-Genesis	20.02	0.218	0.271	30.68	0.2584
	mmGPE	20.09	0.220	0.253	54.22	0.4990
	HybridSim (Ours)	22.29	0.262	0.084	92.07	0.9216

In the cross-subject sim-to-real transfer protocol, HybridSim achieves an exceptional 92.07% HAR accuracy, outperforming prior state-of-the-art mmGPE (54.22%) by an absolute margin of +37.85%.

Ablation Study

1. Rendering Formulation Ablation Comparing the decoupled architecture against a Coupled 3DGS baseline that discards the microfacet BRDF and models both direct and indirect paths using a unified Gaussian patch:

Rendering Formulation	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	HAR Acc (%) ↑	HAR F1-Score ↑	Note
Coupled 3DGS	19.12	0.252	0.099	85.61	0.8492	Lacks explicit BRDF; low-degree SH fails on specular direct peaks
HybridSim (Full Decoupled)	22.29	0.262	0.084	92.07	0.9216	Explicitly decouples direct specular bounce from multipath

2. Noise Floor Injection Strategy Ablation Assessing the impact of background noise modeling on downstream sim-to-real classification transfer:

Noise Injection Strategy	HAR Accuracy (%) ↑	HAR F1-Score ↑	Note
Without Noise (w/o noise)	72.79	0.6790	Overfits to artificially sterile backgrounds
Fixed Constant Noise (w. fixed noise)	90.51	0.9068	Matches baseline noise level; demonstrates architectural strength
Learnable Dynamic Noise (Ours)	92.07	0.9216	Adapts noise floor to match physical hardware noise distribution

3. Multipath Simulation Analysis - Omitting the Indirect Path: Removing the indirect branch reduces HAR performance from 92.07% (F1 0.9216) to 88.96% (F1 0.8874), confirming that while direct LoS carries primary motion cues, site-specific multipaths provide vital contextual cues for sim-to-real alignment. - Two-Bounce Specular Ray Tracing: Replacing 3DGS proxy receivers with classical 2-bounce geometric ray tracing requires 3.44 s/frame (over 70% slower than HybridSim's <1 s/frame). Furthermore, rigid empirical reflection coefficients induce spurious specular artifacts, causing downstream HAR accuracy to collapse to 41.6%.

Key Findings

• PSNR Metric Disconnect in Sparse Radar Signals: Because moving limbs occupy only a small spatial footprint in RD heatmaps while background noise dominates the area, global PSNR is heavily biased toward the noise floor. While HybridSim gains a modest +2.20 dB in cross-subject PSNR over mmGPE (22.29 vs. 20.09 dB), LPIPS plunges from 0.253 to 0.084, and downstream activity recognition jumps by +37.85%. This proves perceptual LPIPS and downstream task accuracy are far more diagnostic than PSNR for radar simulation fidelity.
• Microfacet BRDF is Essential for LoS Scattering: The coupled Gaussian baseline suffers a 6.46% drop in HAR accuracy because low-degree spherical harmonics cannot capture sharp specular peaks at grazing angles, and entangling dynamic motion with static room paths leads to severe overfitting.
• Learnable Noise Floor Closes the Domain Gap: Adding dynamically scaled complex Gaussian noise improves downstream accuracy by nearly 20% over pristine zero-noise simulations, effectively replicating the thermal noise and RF frontend clutter of physical radar hardware.

Highlights & Insights

• Principled Physics-Learning Decoupling: The direct path anchors the propagation delay \(\tau_p(t)\) in exact geometry while modeling material reflectance with a microfacet BRDF. Concurrently, the indirect path substitutes intractable multi-bounce integrals with boundary proxy virtual receivers, eliminating exponential ray-tracing complexity while retaining physically grounded path delays \(d\tau/dt\).
• GCN Training Regularizer with Inference Bypass: Employing graph convolution solely as a training-time regularizer stabilizes high-sensitivity BRDF parameter optimization without incurring inference-time graph message passing that would over-smooth genuine high-velocity micro-Doppler signals.
• Generalizable Cross-Domain Paradigm: Converting complex multi-bounce wave propagation into a three-segment proxy receiver pipeline offers an elegant architectural blueprint applicable to indoor acoustic modeling, sonar simulation, and ultra-wideband (UWB) localization.

Limitations & Future Work

• Site-Specific Scene Overfitting: The current static room representation and proxy wall parameters are optimized for a specific room layout, requiring fine-tuning when migrating to a new indoor environment.
• Simplistic Human Surface Representation: Relying on SMPL meshes neglects the electromagnetic scattering properties of loose clothing textures and fine finger articulations.
• Future Directions: The authors propose integrating generalized neural scene priors to decouple spatial room geometry from RF material responses, enabling zero-shot radar simulation across unseen indoor environments.

Related Work & Insights

• vs mmGPE / RF-Genesis: mmGPE utilizes heuristic physical augmentations, while RF-Genesis integrates ray tracing with generative diffusion models. Both suffer from high computational overhead and multipath distortion. HybridSim is the first single-sensor framework to achieve sub-second physics-neural decoupled simulation, boosting downstream HAR accuracy to 92.07% against their 54.22% and 30.68%.
• vs NeRF2 / RF-3DGS: NeRF2 and RF-3DGS target static RF channel modeling using multi-view or distributed antenna arrays, entangling dynamic target motion with environment echoes. HybridSim is tailored for dynamic single-radar human sensing, disentangling motion kinematics from multipath interference via geometric proxy receivers.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant physical decoupling of direct microfacet inverse rendering and indirect proxy-receiver 3DGS.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive intra/cross-subject evaluation, multi-faceted ablations, downstream sim-to-real transfer, and runtime benchmarks.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous electromagnetic derivations, clear motivation, and well-structured experimental narratives.
• Value: ⭐⭐⭐⭐⭐ Provides a foundational, highly efficient digital twin tool for radar-based human sensing and sim-to-real data scaling.