DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/jolin830/DAP-Net
Area: 3D Vision
Keywords: mmWave radar, action recognition, point cloud network, Doppler effect, heterogeneous multi-source generalization

TL;DR¶

To tackle real-world distribution shifts in mmWave action recognition caused by diverse radar devices and frequency bands, this paper introduces UniMM-HAR, the first large-scale heterogeneous multi-source benchmark, alongside DAP-Net, which leverages Doppler priors for dual-space reparameterization and textual semantic alignment to achieve state-of-the-art cross-source generalization.

Background & Motivation¶

Millimeter-wave (mmWave) radar-based human action recognition (HAR) has emerged as an appealing sensing paradigm due to its inherent privacy preservation, non-intrusive nature, and superior robustness against low-light conditions, environmental occlusion, and severe weather. Unlike visual RGB cameras that expose sensitive identity details or wearable IMU sensors that impose physical burdens on users, mmWave radars capture human dynamic reflections as 3D point clouds with rich spatial geometry and motion cues. Consequently, point cloud-based mmWave HAR has advanced rapidly with the release of several benchmark datasets.

However, existing mmWave point cloud HAR datasets remain constrained in scale and are predominantly collected under homogeneous single-source settings—relying on a single radar device model (such as TI IWR1443 or IWR6843) and an identical operating frequency band (e.g., 60–64 GHz or 76–81 GHz). In practical real-world deployments, such as smart homes or healthcare facilities, sensing systems frequently undergo hardware upgrades, device replacements, or multi-device collaborative deployments. When diverse radar devices capture the same physical action, discrepancies in antenna aperture, frequency-modulated continuous-wave (FMCW) bandwidth, and internal signal processing pipelines induce drastic distribution shifts across point density, spatial layout, Doppler velocity range, and noise profiles.

These distribution shifts represent structural domain variations tightly coupled with hardware parameters rather than simple additive noise. Models trained on single-source data heavily overfit device-specific statistical shortcuts instead of invariant action kinematics, suffering catastrophic accuracy drops when tested on unseen devices. To resolve this bottleneck, the authors construct UniMM-HAR, the first multi-source unified benchmark, and observe that while absolute Doppler velocity magnitudes fluctuate across hardware, their relative spatio-temporal patterns (such as periodic velocity oscillations during arm swings or alternating signs during squats) remain highly consistent across sources. Core idea: exploit action-consistent spatio-temporal Doppler patterns as invariant motion priors to perform sample-adaptive geometric densification via soft quantiles, recalibrate dynamic features in feature space, and establish source-agnostic semantic anchors via a pretrained textual space for robust cross-source recognition.

Method¶

Overall Architecture¶

DAP-Net processes heterogeneous mmWave radar point cloud sequences by combining intra-modal enhancement with cross-modal alignment. The input consists of a sequence of frames, where each point is characterized by 3D spatial coordinates $(x, y, z)$ and a 1D radial Doppler velocity $v$. First, the Dual-space Doppler Reparameterization (D²R) module applies Doppler-guided Geometry Reparameterization (DGR) to suppress static noise and selectively densify dynamic points in geometric space, followed by Motion-aware Feature Recalibration (MFR) to inject global motion priors into feature channels. Next, the reparameterized dense point clouds are fed into a generic point cloud backbone to extract high-level motion representations. Finally, the Text Alignment Module (TAM) projects the extracted features into a shared textual embedding space to compute similarity scores against action category text prototypes, fusing textual similarity with the classifier logits to produce the final prediction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input mmWave Point Cloud Sequence<br/>(x, y, z, Doppler velocity)"] --> B["Doppler-guided Geometry Reparameterization (DGR)<br/>Soft Quantile Partition + Tri-branch Densification"]
    B --> C["Motion-aware Feature Recalibration (MFR)<br/>Dynamic Motion Prior + Channel Affine Modulation"]
    C --> D["Point Cloud Backbone<br/>PointMLP / UST-SSM Global Feature Extraction"]
    D --> E["Text Alignment Module (TAM)<br/>CLIP Action Prototypes + Cosine Similarity Alignment"]
    E --> F["Final Action Recognition Prediction"]

Key Designs¶

1. Doppler-guided Geometry Reparameterization: Adaptive Motion Separation and Controlled Densification

Addressing the severe sparsity, device-induced noise, and extreme density disparities across heterogeneous radar sources, naive uniform duplication amplifies noise, while generative or interpolation strategies produce physically inconsistent artifacts without dense point supervision. To overcome this, DGR develops a two-stage mechanism using Doppler physical cues: First, the Doppler-sorted Soft Quantile (DSQ) replaces fixed velocity thresholds with a learnable quantile parameter $q \in (0, 1)$ to capture relative motion intensity per frame. Given frame points sorted in ascending order of Doppler magnitude, the target rank index is defined as $P_t^{\text{tgt}} = q(P_t - 1)$. Gaussian weights centered at this rank are formulated as: $$w_t^i \propto \exp\left(-\frac{(i - P_t^{\text{tgt}})^2}{2\sigma_t^2}\right)$$ The dynamic motion threshold is calculated via soft aggregation $\tau_t = \sum_{i=1}^{P_t} w_t^i v_t^i$. Soft motion indicators $s_t^i = \sigma((v_t^i - \tau_t)/\gamma)$ are then binarized using a Straight-Through Estimator (STE) during forward passes while keeping backward gradients smooth and differentiable. Second, the Tri-branch Motion-Aware Point Densification (TMPD) divides points into fast ($\mathcal{P}_{\text{fast}}$), slow ($\mathcal{P}_{\text{slow}}$), and raw ($\mathcal{P}_{\text{raw}}$) subsets based on confidence $s_i$. The fast branch, corresponding to salient moving limbs, undergoes controlled $r$-fold duplication ($r=5$) to densify dynamic structures consistently across sources. The slow branch retains quasi-static points without duplication to preserve torso geometry, while the raw branch randomly samples points to reach a standardized target budget $P_{\text{goal}} = 1024$. This selective scheme eliminates cross-source density divergence without hallucinating false artifacts.

2. Motion-aware Feature Recalibration: Doppler Dynamics Guiding Feature Modulation

Standard 3D point cloud architectures designed for LiDAR or depth cameras operate almost exclusively on spatial coordinates $(x, y, z)$, overlooking the subtle kinematic nuances contained in Doppler velocity. To explicitly incorporate dynamic physical priors into feature learning, MFR leverages the high-velocity subset $\mathcal{P}_{\text{fast}}$ identified by DGR to modulate high-dimensional point embeddings. Embeddings of the fast points $\mathcal{F}_{\text{fast}}$ are aggregated via global average pooling into a compact motion summary vector: $$c = \frac{1}{|\mathcal{F}_{\text{fast}}|} \sum_{i \in \mathcal{F}_{\text{fast}}} f_i$$ This vector captures the dominant kinetic state of the action across disparate devices. A lightweight multi-layer perceptron (MLP) then maps $c$ to channel-wise affine scaling $\gamma \in \mathbb{R}^C$ and bias $\beta \in \mathbb{R}^C$: $$\gamma = \text{MLP}_\gamma(c), \quad \beta = \text{MLP}_\beta(c)$$ The recalibrated dense point features $\mathcal{F}$ are produced through channel-wise feature modulation: $$\mathcal{F} = \gamma \odot \mathcal{F}_{\text{out}} + \beta$$ Conditioned on real-time velocity distributions, this affine transformation amplifies motion-sensitive channels while damping static clutter and device-specific RF noise, establishing consistent representations across heterogeneous hardware.

3. Text Alignment Module: Cross-modal Textual Priors as Invariant Semantic Anchors

Under extreme cross-device domain shifts, point cloud representations trained solely on sparse, low-SNR radar signals suffer from feature dispersion and centroid drift. Pretrained natural language models possess structured, invariant semantic manifolds that are completely decoupled from radar sensor characteristics. TAM harnesses this cross-modal prior as an immutable semantic anchor. Specifically, for all $K$ action classes, prompt templates such as "a mmWave point cloud of a person [CLS]" are encoded offline using a frozen CLIP text encoder into category prototypes $F_{\text{text}} \in \mathbb{R}^{K \times C_{\text{text}}}$. The global radar embedding $\tilde{F}$ is mapped via an MLP projection layer into the textual space as $F_{\text{mmw}}$, followed by normalized cosine similarity computation: $$S = \text{Softmax}(F_{\text{mmw}} F_{\text{text}}^\top) \in \mathbb{R}^{K \times K}$$ The diagonal similarity vector $s = \text{diag}(S) \in \mathbb{R}^K$ provides semantic consistency scores that are linearly combined with the backbone classifier logits $z \in \mathbb{R}^K$ to yield the final prediction $\hat{y}$. This cross-modal constraint acts as a powerful regularizer, anchoring multi-source radar embeddings to shared semantic prototypes.

A Worked Example¶

Consider a 32-frame radar sequence depicting a squatting action with an initial frame size of $P=64$: 1. Doppler Partitioning & Densification: At maximum downward velocity, the DSQ module ranks all 64 points by radial Doppler speed, computes an adaptive threshold $\tau_t$ using learnable quantile $q=0.3$, and partitions points into 16 fast-moving leg/torso points and 48 slow background/static points. TMPD applies 5-fold replication to the 16 fast points (yielding 80 points), preserves the 48 slow points, and randomly samples $1024 - 80 - 48 = 896$ points from the raw frame, assembling an aligned cloud of 1024 points. 2. Feature Modulation & Backbone Encoding: The fast point features are average-pooled into a 128-dimensional condition vector $c$. An MLP predicts modulation parameters $\gamma$ and $\beta$, scaling feature channels corresponding to vertical descent. PointMLP then extracts a 512-dimensional global representation. 3. Cross-modal Alignment: The projected global feature is compared with 33 CLIP action text prototypes, yielding a dominant similarity score for "a mmWave point cloud of a person squat", which reinforces the radar classification logit to output a confident, correct prediction.

Loss & Training¶

The entire network is trained end-to-end using standard cross-entropy loss between the fused prediction distribution $\hat{y}$ and ground-truth action label $y$: $$\mathcal{L} = -\sum_{k=1}^K y_k \log \hat{y}_k$$ Experiments are performed on two NVIDIA RTX 4090 GPUs for 150 epochs with batch size 128, initial learning rate 0.01, and weight decay $1 \times 10^{-4}$. PointMLP serves as the default backbone with target point count $P_{\text{goal}} = 1024$ and 5-fold fast point duplication.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on UniMM-HAR under Cross-Subject (C-Sub) and Cross-Set (C-Set) protocols across 10 competitive point cloud architectures, categorized by their original point cloud modalities: mmWave point clouds (MPC), static point clouds (SPC), and temporal point clouds (TPC).

Model	Source Venue/Year	Original PC Type	C-Sub Acc (%)	C-Set Acc (%)
RadHAR	mmNSS'19	MPC	42.49	48.47
FastHAR	CIKM'24	MPC	53.72	61.16
PointNet	CVPR'17	SPC	59.27	70.96
DGCNN	NN'18	SPC	71.70	52.18
PointCLIP	CVPR'22	SPC	67.82	69.82
Clip2point	ICCV'23	SPC	64.17	59.56
PointMLP	ICLR'22	SPC	71.06	78.13
PST-Transformer	TPAMI'22	TPC	63.13	79.70
3DInAction	CVPR'24	TPC	73.43	57.81
UST-SSM	ICCV'25	TPC	71.50	48.40
DAP-Net (Ours)	ECCV 2026	MPC	80.72	81.82

Ablation Study¶

Ablations on UniMM-HAR C-Sub evaluate the contributions of D²R (DGR and MFR) and TAM across diverse point cloud backbones:

Backbone	Module Config	Acc (%)	Gain over Baseline
PointMLP	Baseline	71.06	-
PointMLP	+ DGR (Geometry Reparameterization alone)	76.97	+5.91%
PointMLP	+ MFR (Feature Recalibration alone)	78.76	+7.70%
PointMLP	+ D²R (DGR + MFR)	80.00	+8.94%
PointMLP	+ D²R + TAM (Full DAP-Net)	80.72	+9.66%
UST-SSM	Baseline	71.50	-
UST-SSM	+ D²R	74.00	+2.50%
UST-SSM	+ D²R + TAM	74.94	+3.44%
PST-Transformer	Baseline	63.13	-
PST-Transformer	+ D²R	76.73	+13.60%
PST-Transformer	+ D²R + TAM	78.21	+15.08%

Comparing densification strategies at $P_{\text{goal}} = 1024$: - Repeat Sampling: 71.06% - Super-frame Fusion: 59.72% (-11.34%) - MLP Densification: 43.54% (-27.52%) - Proposed DGR: 76.97% (+5.91%)

Comparing quantile methods: fixed thresholds 0.2, 0.5, and 0.8 achieve 76.46%, 76.14%, and 76.12% respectively, while learnable soft quantile DSQ attains 76.97%.

Key Findings¶

D²R is the dominant performance driver: Integrating D²R brings an 8.94% absolute accuracy boost to PointMLP and a substantial 13.60% jump to PST-Transformer, confirming that Doppler motion guidance bridges the gap between sparse radar data and dense point cloud architectures.
Generative point hallucination collapses on radar signals: Complex MLP generation and multi-frame fusion result in severe performance degradation (-27.52% and -11.34%), as lack of dense supervision creates physically invalid points, whereas Doppler-guided selective replication faithfully preserves radar physics.
Substantial cross-source domain alignment: DAP-Net significantly minimizes domain discrepancy metrics (Centroid Distance and CORAL covariance drift), verifying that the learned feature representations are invariant to radar hardware shifts.

Highlights & Insights¶

Physics-grounded Doppler utilization: Rather than treating Doppler velocity as a mere fourth input coordinate or an ad-hoc filtering heuristic, the method leverages relative Doppler dynamics as intrinsic kinematic anchors to drive both geometric densification and channel modulation.
Parameter-efficient plug-and-play formulation: D²R introduces negligible parameter and FLOP overhead, operating as an adaptable front-end and feature modulator that elevates the performance of diverse backbones (MLPs, State Space Models, and Transformers).
Pioneering multi-source radar benchmark: UniMM-HAR addresses the long-standing limitation of homogeneous radar datasets by consolidating 33 actions and 62 subjects across varying frequencies and hardware models.

Limitations & Future Work¶

Subject and action diversity: While UniMM-HAR represents the largest public benchmark, it compiles 62 subjects; expanding to hundreds of subjects and unconstrained outdoor environments remains an open endeavor.
Multi-radar spatial fusion: The framework currently targets single-radar cross-source adaptation, leaving multi-sensor spatial calibration and cooperative radar network fusion for future investigation.
Future directions: Integrating Doppler-guided representations with multi-modal large language models (MLLMs) could enable zero-shot action grounding under heterogeneous sensing conditions.

vs Dedicated Radar Networks (RadHAR, FastHAR): Prior methods engineered specifically for mmWave radar rely on homogeneous single-source assumptions and perform poorly under multi-source distribution shifts (42%–53% on C-Sub). DAP-Net surpasses 80% accuracy through principled motion-aware reparameterization.
vs Dense Point Cloud Backbones (PointNet, PST-Transformer, 3DInAction): Architectures developed for LiDAR assume uniform surface sampling and struggle directly on sparse, noisy radar clouds. DAP-Net serves as an effective adapter, enabling sophisticated spatiotemporal backbones to unlock their full capacity on mmWave inputs.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Pioneers a heterogeneous multi-source radar benchmark and formulates a well-motivated Doppler dual-space reparameterization scheme.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across 10 baseline networks, cross-subject and cross-set protocols, and detailed component ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp physical motivation and clear methodology.
Value: ⭐⭐⭐⭐⭐ Bridges the critical gap between constrained lab radar experiments and scalable real-world deployments.