WiFi-JEPA: Self-supervised Learning for WiFi-CSI 3D Human Pose Estimation

Conference: ECCV 2026
Paper: ECCV paper page
Area: Human Understanding
Keywords: WiFi sensing, 3D human pose estimation, self-supervised learning, JEPA, simulation-based pre-training

TL;DR

WiFi-JEPA preserves the physical structure of the CSI tensor as (channels C, time T, links L) via CSI-specific tokenization, masks entire Tx–Rx antenna links and lets JEPA predict the masked links' representations in latent space (rather than reconstructing hardware-contaminated raw signals), and supplements pre-training with ray-tracing simulation over randomized geometric primitives (sim-object); on PiW3D it achieves 76.8/93.5 mm single/multi-person MPJPE, beating all WiFi baselines, cuts mean cross-environment error by 48.2% versus the baseline, while SimMIM/MAE/BYOL/MoCoV3 — four vision-native SSL objectives — all degrade below training from scratch under identical backbones and protocols.

Background & Motivation

Cameras fail in three common settings: occlusion by walls or obstacles, insufficient illumination, and privacy or regulatory constraints prohibiting visual capture in sensitive spaces. WiFi CSI is a natural alternative sensing channel — signals penetrate walls, human motion modulates multipath propagation, and CSI is a factored complex-valued measurement over subcarriers (frequency), time, and multiple Tx–Rx antenna links. PiW3D already demonstrated multi-person 3D pose estimation with commodity WiFi (a ~90K-frame dataset, errors around 100 mm), but robust WiFi pose estimation still faces three bottlenecks: fragile cross-domain generalization (transceiver placement or furniture changes degrade accuracy), limited label scalability (CSI pose data depends on camera-based annotation in a few rooms with fixed hardware), and CSI itself being noisy and hardware-dependent. Worse, the common practice reshapes CSI into image-like 2D grids to reuse ViT/CNN backbones: this axis mixing entangles subcarrier, time, and link — three physically distinct axes — so patches frequently cross link boundaries and the amplitude-phase boundary, encouraging reliance on device-specific artifacts over pose-relevant dynamics.

The annotation pipeline is the deeper bottleneck: CSI pose data requires synchronized camera or motion-capture labels, naturally capping collection scale; meanwhile raw CSI carries hardware artifacts (clock offsets, quantization noise) that downstream tasks never need. MAE-style masked reconstruction reconstructs inputs in pixel space, preserving such artifacts; JEPA's latent-space prediction objective naturally suits CSI — prediction targets need not retain task-irrelevant distortions. The core question is: given JEPA for CSI, which physical axis should the masking strategy target? The authors' answer is the link axis — the 9 links are multi-view observations of the same physical scene from different spatial viewpoints, and masking entire links forces prediction from cross-link spatial redundancy, which is exactly what makes 3D spatial structure recoverable. Core idea: build CSI-specific tokenization plus link masking around the (C,T,L) physical structure so JEPA predicts masked representations across links in latent space; complement this with a ray-tracing simulation pipeline over randomized geometric primitives producing annotation-free pre-training data, testing the hypothesis that channel-dynamics diversity matters more than scatterer geometric fidelity.

Method

Overall Architecture

The method has two phases. Phase 1 (self-supervised pre-training): inputs are real CSI (PiW3D, ~90K frames) plus simulated CSI (sim-object, ~90K frames), tokenized via CSI-specific tokenization into T×L=180 tokens (T=20 time steps, L=9 spatial links, with C=60 subcarriers as the channel dimension) plus a separable 2D sinusoidal positional embedding; link masking randomly masks all time steps of 5/9 links at ratio r=0.6, the context encoder processes only the 4 visible links' tokens, the predictor takes context embeddings concatenated with learnable mask tokens to predict target representations, the target encoder processes all tokens and only masked positions serve as prediction targets, minimizing the Smooth L1 loss; the target encoder updates via EMA (cosine momentum 0.996→1.0). Phase 2 (supervised fine-tuning): the pretrained encoder attaches to a PETR-style decoder (5 layers, 256 dims, 8 heads) through a 512→256 linear projection, fine-tuned end-to-end on real CSI with differential learning rates — 0.1× the base rate for the encoder to preserve pretrained features, 1.0× for the decoder and regression heads. The model outputs 14 3D keypoints for N detected persons (N×14×3), Hungarian matching assigns queries to instances, and the decoder refines predictions via cross-attention over encoded CSI features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real + simulated CSI<br/>PiW3D and sim-object"] --> B["CSI-specific tokenization<br/>T×L grid of 180 tokens"]
    B --> C["Link masking<br/>mask 5/9 links, all time steps"]
    C --> D["JEPA latent prediction<br/>Smooth L1 + EMA target encoder"]
    D --> E["PETR decoder fine-tuning<br/>differential LR, end-to-end"]
    E --> F["3D pose output<br/>N persons × 14 keypoints"]

Key Designs

1. CSI-specific tokenization: refuse to flatten time and links into one image

Raw PiW3D CSI has shape \(N_{rx}\times N_{ant}\times T\times N_c = 3\times3\times20\times30\); following [28], amplitude is used directly and phase is denoised with PhaseFi, giving a real-valued tensor of 3×3×20×60 (60 = 30 amplitude + 30 denoised-phase channels). The original approach flattens this into a (60, 180) 2D spectrogram, where 6×6 patch embedding lets patches frequently cross link boundaries and the amplitude-phase boundary, merging physically unrelated components into one token. WiFi-JEPA instead reshapes to (C, T, L) = (60, 20, 9): L=9 spatial links (3 receivers × 3 antennas per receiver) each observe the same physical scene from a different spatial viewpoint; C=60 is the subcarrier channel dimension; T=20 is time. A linear embedding with patch size (1,1) over the (T, L) plane yields T×L=180 tokens, each corresponding to one precise spatio-temporal coordinate. The ablation shows this change alone cuts MPJPE from 111.85 to 97.10 (−14.75 mm).

2. Link masking: let cross-link spatial redundancy define the pretext task

In JEPA, the masking strategy defines the pretext task and therefore dictates the semantic quality of learned representations. Link masking removes entire columns of the (T, L) token grid: at ratio r=0.6 it randomly selects round(0.6×9)=5 of the 9 links as targets, keeps the other 4 as context, and removes all time steps of each masked link simultaneously. The model must predict the complete temporal signal of unseen receiver-antenna pairs from the visible links — multi-view learning intuition: the 9 links observe one physical scene from different spatial viewpoints, and cross-link correlation is precisely the carrier of body-position information. The three alternatives compared (time masking, I-JEPA-style random multi-block masking, per-token random masking) all exploit continuity within a link or local neighborhoods only, never forcing cross-viewpoint inference. In ablation, link masking reaches MPJPE 97.10, beating random masking by 8.04 mm with PCK@50 4.52 pp higher, consistent with the halved cross-environment error.

3. Latent-space prediction objective: no raw-CSI reconstruction, no hardware artifacts

MAE/SimMIM-style masked reconstruction must reproduce raw CSI values including hardware artifacts (clock offsets, quantization noise), biasing the encoder toward device-specific patterns. WiFi-JEPA predicts in latent space: the context encoder encodes visible tokens, the predictor predicts target-encoder representations at masked positions, with loss

\[\mathcal{L} = \frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\left[\text{SmoothL1}\left(p_\phi(x_{\mathcal{C}}, p_{\mathcal{M}})_i,\ \text{sg}[\text{LN}(\bar{f}_\theta(x)_i)]\right)\right],\]

where \(p_{\mathcal{M}}\) are positional embeddings at masked locations, \(\text{sg}[\cdot]\) is stop-gradient, and LN applies to target representations; the target encoder \(\bar{f}_\theta\) updates via EMA (momentum 0.996→1.0 cosine schedule). Context and target encoders share one ViT backbone (12 layers, 512 dims, 8 heads) while the predictor is a shallower ViT (8 layers, 512 dims, 16 heads) — limiting predictor capacity is the classic JEPA device, forcing the context encoder to learn transferable, pose-relevant features rather than outsourcing the prediction task to a high-capacity predictor.

4. sim-object pipeline: geometric primitives, not human meshes, manufacture pre-training data

Large-scale CSI collection requires dedicated indoor environments, synchronized receivers, and co-located motion capture for labels; the authors bypass this with a two-stage simulation. Stage 1 (pure Python, CPU-only, <1 s per clip) generates randomized indoor scenes: room sides 3–8 m, height 2.5–4 m, ITU-standard wall materials (concrete/plasterboard/wood/glass), 1–4 geometric primitives (sphere/cube/cylinder/ellipsoid, radius 0.1–0.5 m) moving along physics-based trajectories (initial velocities up to 3 m/s — exceeding typical indoor speeds — with elastic wall reflections). The transceiver configuration matches the target benchmark's SIMO setup (1 Tx + 3 receivers × 3 antennas = 9 links). Stage 2 uses NVIDIA Sionna RT for ray tracing with one critical choice: 20 independent ray-tracing passes per frame. Each pass computes the channel for a static snapshot, sampling object positions at 20 sub-frame locations within the ~50 ms measurement window — without it, a single pass yields a near-constant time axis and temporal masking becomes ineffective. The output \(H\in\mathbb{C}^{20\times30\times3\times3}\) decomposes into amplitude plus phase, yielding the real-valued (60, 20, 9) tensor for tokenization. 90K frames cost ~10 GPU-hours on one RTX 4090. At the hypothesis level: at 5.64 GHz (wavelength ≈5 cm) a sphere scatters very differently from a human body, but the authors bet that SSL needs diverse spatio-temporal channel variation, not faithful reproduction of body-specific multipath — tested directly by pitting sim-object against sim-human.

Loss & Training

Pre-training runs 100 epochs with AdamW (batch 64, lr 5×10⁻⁴, weight decay 0.05), Smooth L1 on layer-normalized target representations; masking ratio fixed at 0.6. Fine-tuning runs 100 epochs with AdamW (base lr 2×10⁻⁵, weight decay 10⁻⁴), encoder lr scaled to 0.1×. Two pre-training data configurations: real-only (PiW3D 90K frames) and real+simulated (90K+90K); all training fits on a single RTX 4090, with simulation generation costing an additional ~10 GPU-hours.

Key Experimental Results

Main Results

PiW3D is the only public WiFi-CSI benchmark with multi-person 3D pose annotations: three indoor environments (office/classroom/corridor, each ~4×3.5 m), 1 Tx + 3 Intel 5300 receivers (3 antennas each) at 5.64 GHz with 30 subcarriers; the training set has 89,946 frames and the test set 7,824 frames spanning single-person (2,586), two-person (3,184), and three-person (2,054) scenarios. Metrics are MPJPE (primary — mean Euclidean distance between predicted and ground-truth 3D joints), PA-MPJPE (Procrustes-aligned, removing global translation/rotation/scale to isolate articulated pose accuracy), and PCK@20/50. Of the five baselines compared, WiPose/MetaFi++/HPE-Li/DT-Pose report no multi-person results († values taken from original publications).

The table below excerpts the original Table 1 (PiW3D, single/multi-person):

Method	SP MPJPE↓	SP PA↓	SP PCK@20↑	MP MPJPE↓	MP PCK@20↑
WiPose (MobiCom'20)	101.8	—	—	—	—
MetaFi++ (IoT-J'23)	132.0	75.8	62.0	—	—
HPE-Li (ECCV'24)	120.2	69.5	59.1	—	—
DT-Pose (arXiv'25)	90.0	58.7	72.1	—	—
PiW3D baseline (CVPR'24)	91.7	55.1	69.3	107.2	58.1
WiFi-JEPA (real)	78.2	53.9	74.5	97.1	59.3
WiFi-JEPA (real+sim)	76.8	54.0	75.9	93.5	61.5

Single-person MPJPE improves on the previous best DT-Pose (90.0) by 13.2 mm (−14.7%), and multi-person drops 13.7 mm from PiW3D (107.2) to 93.5 (−12.8%); the real-only pre-training variant (78.2/97.1) already beats all baselines, and adding simulation shaves a further 1.4/3.6 mm. Cross-environment (original Table 2, leave-one-environment-out): both pre-training and fine-tuning exclude the held-out environment; WiFi-JEPA scores 248.4/428.2/296.0 mm on Office/Classroom/Corridor, mean 324.2 mm — a 48.2% reduction from the PiW3D baseline's 626.4 mm; in the few-shot setting with 10% of target-environment labels it beats Supervised+DANN (171.3 vs 208.6 mm, −17.9%). Per person count (original Table 3): 1P 76.8, 2P 96.0, 3P 110.7, degrading gracefully. Extremity breakdown (original Table 4): the baseline's hardest joints, elbows/hands, improve from 160.5 to 64.7 mm (−59.7%), with hand depth error dropping from 92.9 to 26.4 mm.

Ablation Study

The table below excerpts the original Table 5, pre-training data comparison (all rows share the same encoder, decoder, and fine-tuning protocol; only pre-training data varies; Δ relative to from-scratch 102.4 mm):

Pre-training data	Real frames	Sim frames	MPJPE↓	Δ
None (scratch)	—	—	102.4	—
sim-object	—	90K	100.1	−2.3
sim-human	—	90K	110.3	+7.9
Real only	90K	—	97.1	−5.3
Real + sim-object	45K	45K	97.2	−5.2
Real + sim-object	90K	90K	93.5	−8.9

The table below combines the original Tables 6, 7, and 8 (same ViT backbone and PETR-style decoder, pretrained on ~90K real frames for 100 epochs, identical fine-tuning):

Comparison	MPJPE↓	PCK@50↑
SimMIM (MIM)	145.6	84.3%
MAE (MIM)	130.3	87.4%
BYOL (self-distill)	144.4	84.1%
MoCo v3 (contrastive)	141.5	84.9%
WiFi-JEPA w/o pretrain	102.4	91.8%
WiFi-JEPA w/ pretrain	97.1	93.0%
Masking — random	105.14	88.48%
Masking — multi-block (I-JEPA)	102.06	90.54%
Masking — time	104.24	88.22%
Masking — link	97.10	93.00%

Tokenization ablation (original Table 8): CSI-specific tokenization (60×20×9) versus the 2D spectrogram (1×60×180, 6×6 patches) reduces MPJPE 111.85→97.10 (−14.75 mm) and PA-MPJPE 70.98→67.20 (−3.78 mm).

Key Findings

• All four vision-native SSL objectives transfer negatively. SimMIM/MAE/BYOL/MoCo v3 pretrained results (130.3–145.6 mm) all fall below from-scratch training at 102.4 mm — MIM is forced to reproduce hardware artifacts, and the augmentations behind self-distillation/contrastive learning lack CSI physical grounding; only the physics-structure-aware JEPA improves consistently (−5.3 mm).
• Dynamics diversity beats geometric fidelity. Geometric primitives (sim-object 100.1) beat human meshes (sim-human 110.3) by 10.2 mm, and sim-human transfers negatively (+7.9); fixed motion-capture sequences lack trajectory variation, whereas randomized physics trajectories (velocities, elastic reflections) drive generalizable channel features.
• Simulated and real data are complementary, not substitutes. sim-object alone (100.1) approaches real-only (97.1) within 3.0 mm; combining them extends the improvement from −5.3 to −8.9 mm, with simulation supplying room-geometry and wall-material variation absent from the real data.
• Cross-environment remains far worse than same-environment. The LOO mean of 324.2 mm halves the baseline but is still high in absolute terms — link masking buys environment robustness without closing the domain gap.
• Gains concentrate in global localization, not joint configuration. PA-MPJPE stays close to the from-scratch baseline (67.2 vs 65.1); the large MPJPE drop comes mainly from improved global position estimates.

Highlights & Insights

• Define token boundaries along physical axes. (C, T, L) structured tokenization makes every token a precise spatio-temporal coordinate, eliminating cross-boundary patches; the principle applies to any factored signal — radar, ultrasound, microphone arrays — not just CSI.
• Masking strategy as inductive bias. Masking whole links = forcing cross-viewpoint inference, writing "multi-link = multi-view" — a structure unique to CSI — into the pretext task, carrying more signal than generic random or block masking.
• The division of labor: simulate for pre-training, real for fine-tuning. Simulation contributes only annotation-free channel-variation diversity, while final fine-tuning aligns with the hardware distribution on real data; the 20 RT passes per frame are the necessary design that makes the simulated time axis genuinely vary.
• Latent-space objectives fit denoising naturally. JEPA never requires preserving hardware distortions in raw signals — its core fit for CSI; the systematic failure of the four vision SSLs conversely supports the "modality-native SSL" direction.

Limitations & Future Work

• Single dataset, fixed hardware. Evaluation covers only PiW3D (1 Tx, 3×3 Intel 5300, 5.64 GHz); generalization across antenna configurations and frequency bands is unverified.
• No PA-MPJPE improvement. Global localization improves substantially while relative joint-configuration accuracy stagnates — the authors acknowledge this.
• Cross-environment absolute error remains high. The LOO mean of 324.2 mm is still far from usable; link masking is necessary but not sufficient.
• The simulation-domain gap is not decomposed. Channel-distribution differences between sim-object and real data (material electromagnetic parameters, antenna-pattern modeling error) are not quantified separately; "dynamics diversity matters more" holds within the simulated side's comparison (vs sim-human) but is not tested against real human subjects.
• No single/mixed-person training ablation. The paper does not report person-count-specific training variants, so multi-person gains may partly reflect the training distribution.

Related Work & Insights

• vs DT-Pose: the only prior WiFi pose SSL method, combining MAE pre-training, temporal contrastive learning, uniformity regularization, and a GCN–Transformer decoder with skeleton-topology constraints; this paper swaps the pre-training objective for JEPA and tokenizes along physical axes, and controls decoder architecture to isolate the pre-training objective's effect (DT-Pose carries a decoder inductive bias that the PETR decoder here deliberately lacks).
• vs PiW3D: the first multi-person 3D WiFi benchmark and supervised baseline; this paper reuses its data and evaluation protocol, showing SSL pre-training layers directly onto a comparable decoder.
• vs MAE / SimMIM / BYOL / MoCo v3: all transfer negatively under identical backbones and data, showing that transplanting image SSL objectives to CSI fails unless the masking structure matches the physical axes.
• vs CIG-MAE / AM-FM / SSLCSI: existing CSI-SSL works either borrow image augmentations/masking or allocate masking by patch information density, but none distinguishes CSI's three physical axes and none tackles multi-person 3D pose; WirelessJEPA operates on raw IQ streams for communication/RF classification, a different input and objective from estimated CSI for sensing.
• vs FractalDB / Dead Leaves: precedents for non-semantic visual pre-training; this paper ports "structural diversity beats geometric realism" into the wireless-channel domain and verifies it with the sim-human control.

Rating

• Novelty: 4/5. The combination of CSI-specific tokenization, link masking, and ray-tracing simulation pre-training is new, with direct experimental support for the "dynamics diversity beats geometric fidelity" hypothesis; JEPA itself is not new.
• Experimental Thoroughness: 4/5. Main table, LOO cross-environment, DANN comparison, per-person-count and per-joint breakdowns, plus three ablation groups (SSL objectives, masking, tokenization, pre-training data) give broad coverage; the single-dataset scope is the main limitation.
• Writing Quality: 4/5. The three-question-driven experiment design is clear and hypothesis testing is explicit; cost accounting between simulation and real data is slightly mixed.
• Value: 4/5. Directly useful for pose sensing in camera-denied scenarios; the simulation pipeline (~10 GPU-hours for 90K frames) is reusable as a pre-training data source for other WiFi sensing tasks.