Skip to content

MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots

Conference: ECCV 2026
Paper: ECCV Official
Code: https://autonomousrobots.nl/paper_websites/mobileocc
Area: Autonomous Driving
Keywords: 3D semantic occupancy, mobile robot, human mesh optimization, occupancy prediction benchmark, pedestrian velocity prediction

TL;DR

Built on synchronized images and LiDAR from the UT Campus Object Dataset (CODa), MobileOcc is a semantic occupancy dataset for mobile robots in pedestrian-dense environments: deformable human meshes obtained by jointly optimizing a monocular SMPL initialization against LiDAR points replace rigid bounding-box proxies, covering 116,511 frames at up to 0.02 m label resolution, together with a benchmark suite for occupancy prediction and pedestrian velocity prediction. Static voxel labels reach 71.2 mIoU and pedestrian voxels reach 92.5 F1 against a LiDAR body-surface reference, while existing occupancy models attain only 26.6-29.9 mIoU on the benchmark.

Background & Motivation

Dense 3D semantic occupancy prediction has become a standard component of modern navigation stacks: methods such as VoxFormer, OccDepth, TPVFormer, SurroundOcc, FlashOcc and Panoptic-FlashOcc infer a voxel field directly from images, assigning each voxel a free / occupied / unknown state plus a semantic label, which yields a differentiable perception representation that plugs into downstream planning. Yet the annotated datasets that train these models are almost entirely rooted in autonomous driving. SemanticKITTI, Occ3D, OpenOccupancy and SSCBench build on KITTI, nuScenes and Waymo, while WildOcc covers off-road terrain. Their shared bias is a rigid world: roads, buildings and vehicles are the protagonists, and dynamic objects are accumulated into rigid bodies using pre-labeled 3D bounding-box poses. Pedestrians are rare in these datasets and are treated as rigid bodies, which erases their non-rigid shape.

This is a real problem for mobile robots. When a robot navigates sidewalks, campuses or shopping malls, humans are not background clutter but the entities it deals with most often, and what determines the occupied shape of a person is precisely the limbs: a raised arm, a leg mid-stride, the posture of someone carrying a box. Approximating a person by a bounding box declares the body plus a shell of surrounding air to be occupied; in dense crowds, that volumetric inflation directly translates into passages judged non-traversable or into socially unsafe trajectories, and it is incompatible with physical human-robot interaction that needs limb-level, pose-dependent representations. A second gap is identity: driving occupancy datasets largely lack instance tracking (only something like OpenScene provides it), whereas crowd navigation cares not only about whether a place is occupied but also about whether that occupancy belongs to one and the same person and where that person is heading - which requires per-instance identities and a way to evaluate velocity.

This paper therefore takes a gap-filling route: rather than improving an occupancy network, it supplies the missing artifact. What makes this feasible now is that robot platform datasets such as CODa already provide long, time-synchronized camera-LiDAR streams, and two lines of work have matured independently - image-based human mesh recovery (HMR) with SMPL-type models, and LiDAR-based human fitting. Neither suffices alone. Purely image-based HMR (CLIFF, ROMP, HybrIK and others) suffers from depth and scale ambiguity: the pose looks right in the camera frame while the absolute placement cannot be trusted. Purely LiDAR-based fitting (LiDARCap, LiDAR-HMR, VoteHMR) depends on dataset-specific training and becomes brittle when human points mix with nearby object points or when points thin out at range, causing missed detections and uncertain pose. Core idea: let images supply a reliable initial pose and shape while LiDAR supplies absolute placement and body-surface constraints, stitching the two together with an optimization that is carried out at inference time and requires no training; then fuse the resulting deformable human meshes with a static semantic map from which pedestrian evidence has been explicitly removed, arbitrating between them in the robot frame under a dynamic-over-static priority rule to produce voxel occupancy labels with per-instance IDs - and package occupancy prediction together with pedestrian velocity prediction into a reproducible benchmark with metrics and baseline implementations.

Method

Overall Architecture

The pipeline consumes synchronized RGB image streams, LiDAR sweeps and trajectory poses collected by the CODa robot on a campus, and emits semantic occupancy voxels in the robot-local frame (up to 0.02 m resolution, written out in nuScenes format so existing tooling can consume it directly). It proceeds in three steps. First, a cross-modal preprocessing pass aligns everything into "pedestrians with IDs plus 3D points with semantics". Then the work splits into two parallel branches: one turns pedestrians into deformable meshes (human mesh optimization), the other turns the rest of the world into a human-free static semantic map (static semantic map construction). Finally the two branches merge in the shared robot-local frame, where a priority rule decides which claim wins for each voxel (occupancy label assembly), yielding a voxel grid in which static semantics and dynamic human occupancy coexist. That grid is usable downstream as-is, and is also downsampled to a 0.2 m evaluation grid that defines the two tasks and their baselines (dual-task benchmark).

The preprocessing itself is not this paper's contribution, but everything downstream depends on it, so it is worth stating plainly: YOLOX detection with OC-SORT tracking yields 2D pedestrian boxes with IDs; for each tracked person, ViTPose estimates 2D keypoints with confidences; Mask2Former produces instance masks that are associated with the LiDAR point cloud (this is what decides which 3D points belong to which person); and static background elements are labeled with Cityscapes semantic segmentation. All preprocessing outputs are time-stamped and co-referenced across modalities, and both mesh optimization and static mapping run on that shared temporal-spatial basis.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized images + LiDAR + robot poses"] --> B["Preprocessing: tracking / 2D pose<br/>instance masks / semantic segmentation"]
    B --> C["Human mesh optimization<br/>visibility filter → ICP → joint optimization"]
    B --> D["Static semantic map construction<br/>pedestrian suppression → multi-frame fusion → OctoMap"]
    C --> E["Occupancy label assembly<br/>dynamic-over-static arbitration + instance IDs"]
    D --> E
    E --> F["Semantic occupancy labels at 0.02 m<br/>downsampled to 0.2 m for the benchmark"]
    F --> G["Dual-task benchmark<br/>occupancy prediction + pedestrian velocity prediction"]

Key Designs

1. Human mesh optimization: LiDAR body-surface constraints remove the depth and scale ambiguity of monocular HMR

As noted above, the SMPL mesh produced by monocular HMR looks plausible in the camera frame but its absolute depth and scale cannot be trusted, while LiDAR-based methods have the opposite profile: accurate absolute placement, but missed detections and drifting pose when human points mix with background points or thin out at distance. This design takes both. It starts from an image-based initial estimate and pulls it to the true 3D position and posture using the point cloud, in three stages, and it trains no network and uses no dataset-specific supervision, which is why it transfers to other datasets unchanged.

The first stage is visibility filtering. Given the initial parameters \(\beta_0, \theta_0, t_0\) predicted by CLIFF (a top-down HMR regressor that carries full-frame location information into the person crop), keypoint confidences indicate which body parts are in fact occluded in the image, and only the visible vertex subset \(\mathcal{V}\subset M\) is kept, where the mesh is \(M(\beta,\theta)\in\mathbb{R}^{6890\times3}\). The point of this step is that an occluded elbow or a leg hidden behind the torso simply has no image evidence; forcing it to align with LiDAR would drag the pose away, so those parts are flagged as invisible and handled conservatively later. The second stage is coarse alignment: ICP registers the visible vertices \(\mathcal{V}\) against the LiDAR cloud \(\mathcal{P}\) to obtain a rigid transform \(T\in SE(3)\), initialized from \((\theta_{\text{glob}}, t_{\text{cam}})\), producing an updated global orientation and translation \((\theta'_{\text{glob}}, t'_{\text{cam}})\) - this answers "where is this person". ICP is rigid and leaves joint angles untouched, so the aligned mesh may no longer be consistent with the image pose (the inconsistency circled in red in the paper's Fig. 2). The third stage is therefore a joint optimization that frees the shape \(\beta\), the pose \(\theta\) and the camera translation \(t_{\text{cam}}\) to fit the 2D keypoints and the LiDAR points simultaneously under human priors, minimizing \(\mathcal{L}_{\text{total}}(\beta,\theta,\mathbf{t}_{\text{cam}})\) (terms below). The division of labor is clean: visibility filtering decides what to trust, ICP decides where, and joint optimization decides in what posture.

Compared with the classical single-image fitting recipe of SMPLify, this work drops the interpenetration term and instead adds a LiDAR-mesh Chamfer alignment term and an occlusion-aware pose consistency term, keeping 2D reprojection, the MoG pose prior, the anti-hyperextension prior and the shape prior. The reason for dropping interpenetration is concrete: it is expensive and contributes little to final accuracy; more importantly, occupancy here is voxelized from the mesh surface, so interpenetration only creates rare internal self-overlap and barely moves the occupied/free boundary, so paying for it is not worth it. The occlusion-aware term, by contrast, targets a genuine failure mode - hidden limbs have no LiDAR evidence and are exactly the parameters an optimizer will drag to absurd values.

2. Static semantic map construction: erase pedestrian evidence explicitly before multi-frame fusion

The static semantic map is the background layer of every occupancy label, and its correctness has a subtle enemy: if pedestrian points are accumulated as ordinary geometry during mapping, they leave ghosts in the background. Those cells are in fact empty in the vast majority of frames, and because the static map is shared across the sequence, one ghost propagates across frames and viewpoints into an obstacle that never existed. Worse, pedestrians the camera cannot see (outside the frustum, behind buildings) cannot be excluded by image detection at all.

The paper attacks this from two sides. The first is pedestrian evidence suppression: a LiDAR 3D detector runs on every sweep, and its pedestrian outputs are combined with the image-based pedestrian detections into a mask; points falling inside that mask are excluded from static map generation. The value of the LiDAR path is precisely that it covers the blind spots outside the camera frustum. This stage changes no semantic labels and trains no model - its sole purpose is to suppress pedestrian evidence before mapping. The second is multi-frame aggregation: each sweep is pose-compensated, camera-derived semantics are lifted onto the corresponding 3D points, and those points are inserted into a voxel map that maintains per-voxel label counts; as frames accumulate, voxel labels are updated by max-voting while geometric sanity checks such as ground consistency reject outliers. Once the sequence is processed, each voxel receives a definite semantic label, and the remaining empty cells are filled with free / unknown states via an OctoMap query. The result is a human-free, high-resolution static semantic background onto which human occupancy is overlaid separately - the two responsibilities are separated, so failures in one do not contaminate the other.

3. Occupancy label assembly: dynamic-over-static label arbitration with per-instance IDs

With a human-free static background and per-frame human meshes in hand, the last step is to fuse them into a self-consistent occupancy grid. For each frame, the current static points and the rasterized human meshes are transformed into a common local coordinate frame using the synchronized robot pose and then voxelized. The actual design lies in the priority order of label arbitration: dynamic humans take precedence over static background. This rule looks simple but covers two engineering hazards at once. First, even if the masking stage misses a pedestrian, the human mesh will still claim that voxel back from the background. Second, precisely because the static map was built with pedestrian suppression and human occupancy overrides static labels during alignment, the effect of a missed detection is confined to the frame where it happens instead of spreading into a global error through the static map. For voxels with no pedestrian in them, the majority-label-wins rule fixes the label, which avoids label oscillation when several points with different labels land in the same voxel. Any remaining empty cells are completed by an OctoMap query so that occupied space is always supported by explicit geometry.

Being "human-aware" lands in two concrete places here rather than in extra body-part classes. The first is geometry: a pedestrian is not a bounding box but a set of voxels filled by a per-frame optimized SMPL mesh, so limb-level shape survives at the 0.02 m fine resolution (the paper's Fig. 1 shows exactly this contrast, fine versus coarse resolution for the same scene). The second is identity: every tracked pedestrian is assigned a unique ID, so pedestrian voxels carry per-instance labels and static semantics coexist with dynamic human occupancy in a single grid. The last items are resolution and format: grid resolution is user-specified, up to 0.02 m, and is uniformly downsampled to 0.2 m for the benchmark, while outputs follow the nuScenes dataset format for easy integration with existing tools.

4. Dual-task benchmark: bringing driving occupancy models into monocular, stereo and panoptic settings

The dataset is only half the work; a reproducible evaluation protocol is the other half. Evaluation runs on a 0.2 m voxel grid spanning \(x\in[0.4,10.0]\) m, \(y\in[-4.8,4.8]\) m and \(z\in[-1.0,3.8]\) m at 5 Hz, with the taxonomy collapsed to 10 classes (one free-space plus nine occupied classes), merging bicycle, motorcycle and scooter into two-wheeler and several static structural categories into other-structure to reduce long-tail noise. The training and validation splits contain 92,303 and 24,208 frames (79/21%), of which 30,457 and 7,165 carry pedestrian annotations.

Baseline adaptation is the most practical part of this section, because the input distribution of MobileOcc differs from that of driving datasets: BEVDet4D, FlashOcc and Panoptic-FlashOcc all assume multi-view surround cameras, whereas this dataset only has front-view images, so they are adapted to a monocular setting using only the left stereo camera for depth estimation, with the 8 historical frames of their public settings. VoxFormer-T takes the stereo route instead: 4 historical frames sampled alternately in time to match the temporal coverage of the 8-frame monocular baselines, with stage-1 consuming voxelized stereo-matching depth to predict a 3D occupancy volume at 0.4 m, which is then upsampled to the 0.2 m target grid. The paper additionally introduces Panoptic-FlashOcc-vel, which extends the detection head of Panoptic-FlashOcc with two channels predicting \((v_x,v_y)\) under an L1 loss. Its purpose is to let a single forward pass deliver voxel semantics, instance assignment and pedestrian velocity at once, instead of maintaining a dedicated velocity model - which is exactly the motivation "a mobile robot needs to know where people are going" made concrete at the evaluation level.

For metrics, occupancy prediction uses IoU and mIoU: the per-class IoU is the intersection-over-union of predicted and ground-truth occupied voxel sets, \(|P\cap G|/|P\cup G|\), and mIoU averages it over the nine occupied classes (hence it is unaffected by classes absent from the ground truth). Panoptic occupancy uses the standard PQ / RQ / SQ and additionally reports PQ-dagger, a variant that relaxes the strict IoU requirement for stuff classes and suits vision-only occupancy prediction better. Pedestrian detection uses AP\(_{\text{Ped}}\), computed on detected centers across distance thresholds of 0.1 m, 0.2 m, 0.5 m and 1 m. Velocity prediction uses three absolute velocity errors: AVE-T over all ground-truth pedestrian voxels, AVE-D over true-positive detections within 1 m, and AVE-O over correctly classified voxels occupied by pedestrians. On the human mesh side the metrics are PVE (average per-vertex Euclidean distance), MPJPE (mean per-joint position error), PA-MPJPE (Procrustes-aligned MPJPE, which strips out global placement and isolates articulated pose quality) and MPERE (mean per-edge relative error, reported only when vertex ground truth exists or the method outputs a mesh).

Loss & Training

Human mesh optimization is the only part of the paper with an objective function, and it is optimized at inference time: mesh parameters are solved per tracked pedestrian at test time, with no network training and no dataset-specific LiDAR supervision, which is exactly why the method transfers directly to 3DPW, SLOPER4D and HumanM3. The total objective sums image alignment, 3D alignment and three human priors with scalar weights:

\[\mathcal{L}_{\text{total}} = \lambda_J \mathcal{L}_J + \lambda_{3D}\mathcal{L}_{3D} + \lambda_\theta \mathcal{L}_\theta + \lambda_a \mathcal{L}_a + \lambda_\beta \mathcal{L}_\beta + \lambda_{\text{occ}}\mathcal{L}_{\theta,\text{occ}}\]

⚠️ The extracted Eq. (1) is incomplete (the weight of the 2D reprojection term was lost); the form above follows the prose description of the individual terms - refer to the original paper.

The 2D joint reprojection term projects the 3D joints of the mesh back into the image and measures their distance to detected 2D keypoints, weighting each keypoint by its confidence \(w_i\) and using the Geman-McClure robust penalty \(\rho(\cdot)\) to suppress outliers:

\[\mathcal{L}_{J} = \sum_i w_i\,\rho\Big(\Pi_{K}\big(\mathbf{R}(\theta)\mathbf{J}_i(\beta) + \mathbf{t}_{\text{cam}}\big) - \mathbf{J}_i^{\text{est}}\Big)\]

The 3D alignment term uses a symmetric, size-normalized Chamfer distance between visible mesh vertices and the LiDAR cloud, measuring nearest-neighbor distances in both directions; because it is differentiated with respect to \(\theta\) and \(\beta\), it can correct local pose and shape errors that pure 2D fitting cannot reach:

\[\mathcal{L}_{3D} = \frac{1}{|\mathcal{V}|}\sum_{\mathbf{v}\in\mathcal{V}}\min_{\mathbf{p}\in\mathcal{P}}\|\mathbf{v}-\mathbf{p}\|_2^2 \;+\; \frac{1}{|\mathcal{P}|}\sum_{\mathbf{p}\in\mathcal{P}}\min_{\mathbf{v}\in\mathcal{V}}\|\mathbf{p}-\mathbf{v}\|_2^2\]

The remaining three priors are: the MoG pose prior \(\mathcal{L}_\theta\), a negative log-likelihood over a Gaussian mixture of plausible joint configurations that discourages unnatural joint combinations; the anti-hyperextension prior \(\mathcal{L}_a\), which penalizes unnatural positive bending at elbows and knees; and the shape prior \(\mathcal{L}_\beta\), a quadratic penalty on \(\beta\) that keeps the shape inside the SMPL space. One further term targets occlusion, \(\mathcal{L}_{\theta,\text{occ}}\): for the set of joints \(\mathcal{I}\) flagged invisible by the visibility filter, it penalizes deviation from the initialization in terms of the unit quaternions (\(\mathbf{q}_i^{(0)}\) versus \(\mathbf{q}_i\)), preventing joints that have no observational evidence from drifting. ⚠️ The exact functional form (how the quaternion deviation is measured and whether it is squared) is corrupted in the extracted text - refer to Eq. (7) of the original paper.

The optimized variables are \((\beta,\theta,\mathbf{t}_{\text{cam}})\), solved by gradient descent in PyTorch to yield \(\hat\beta,\hat\theta,\mathbf{t}^*_{\text{cam}}\) that balance fitting the image against fitting the point cloud while respecting human shape and pose priors. The benchmark side introduces no new training strategy: every occupancy baseline keeps its public training settings with 8-frame history (4 alternately sampled frames for VoxFormer-T), and the velocity head of Panoptic-FlashOcc-vel is trained with an L1 loss.

Key Experimental Results

Main Results

Evaluation covers four blocks: human mesh optimization (on 3DPW, SLOPER4D and HumanM3), occupancy label quality (against CODa ground truth), detection robustness, and the two tasks after training baselines on MobileOcc.

Table 1: 3D semantic occupancy prediction on MobileOcc (0.2 m voxels; IoU is geometric IoU; best in bold)

Method IoU mIoU ped. car oth.str. pole road terrain truck 2-wh. veg.
VF-T (4f stereo) 57.81 24.89 32.79 1.39 28.47 7.08 70.90 23.57 5.23 28.04 26.52
FO (8f mono) 57.71 27.18 31.91 5.36 31.00 10.16 70.82 27.29 7.35 30.85 29.88
PF (8f mono) 57.83 26.64 32.45 3.66 31.79 9.96 70.35 25.90 5.87 30.99 28.83

VF-T = VoxFormer-T, FO = FlashOcc, PF = Panoptic-FlashOcc.

Table 2: Panoptic occupancy and pedestrian velocity prediction (BD = BEVDet4D, PF-vel = Panoptic-FlashOcc-vel)

Method PQ PQ-dagger RQ SQ PQ\(_{\text{Ped}}\) RQ\(_{\text{Ped}}\) SQ\(_{\text{Ped}}\) AP\(_{\text{Ped}}\) AVE-T ↓ AVE-D ↓ AVE-O ↓ mIoU
BD (8f) - - - - - - - 41.7 1.00 0.36 - -
PF (8f) 19.9 32.6 28.1 65.8 42.5 60.2 70.7 45.5 - - - -
PF-vel (8f) - - - - - - - - 0.97 0.39 0.67 26.00

Ablation Study

A dataset paper has no conventional module ablation; in its place come two groups of controls: how mesh accuracy changes with sensor density once LiDAR is added, and whether it still holds on real point clouds; plus whether the labels themselves are correct, measured against ground truth, against a tighter reference, and against upstream detection errors.

Table 3: Simulated-LiDAR comparison on 3DPW (image and video methods have no depth and are reported with root alignment; ours fuses simulated LiDAR without root alignment)

Modality Method PVE ↓ MPJPE ↓ PA-MPJPE ↓
Video VIBE 99.1 82.9 51.9
Video MotionBERT 79.4 68.8 40.6
Video WHAM-B 71.0 59.4 37.2
Image CLIFF 81.2 69.0 43.0
Image PLIKS 73.3 60.5 38.5
LiDAR Ours (Ouster-32) 73.2 57.0 47.7
LiDAR Ours (Ouster-64) 57.2 43.9 38.5
LiDAR Ours (Ouster-128) 50.5 39.1 35.1

Table 4: Generalization on real LiDAR (SLOPER4D and HumanM3; the PA-MPJPE columns isolate the net gain of adding LiDAR optimization on top of the CLIFF initialization)

Method Modality SLOPER4D PVE ↓ SLOPER4D MPJPE ↓ SLOPER4D PA-MPJPE ↓ SLOPER4D MPERE ↓ HumanM3 MPJPE ↓ HumanM3 PA-MPJPE ↓
PRN LiDAR - 57.0 - - 82.2 -
V2V-PoseNet LiDAR - 50.7 - - 83.0 -
LiDAR-HMR LiDAR 51.9 51.0 - 0.094 77.6 -
LiDARCap LiDAR 148.1 158.3 - 0.050 175.8 -
SAHSR LiDAR 81.2 72.6 - 0.085 105.5 -
VoteHMR LiDAR 60.9 54.6 - 0.079 105.8 -
CLIFF RGB 93.7 84.7 69.3 0.057 106.3 66.5
Ours RGB+LiDAR 64.4 55.8 43.5 0.060 83.9 58.1

Table 5: Occupancy label quality and upstream detection robustness

Evaluation Metric Value
Static voxels (vs CODa GT points, 4 sequences, 400 frames) mIoU (road / veg. / terrain) 71.2 (88.8 / 68.0 / 57.0)
Static voxels GT point coverage / voxel-level accuracy / point-level accuracy 96.8 / 88.1 / 94.0
Pedestrian voxels (vs GT 3D bounding-box proxy) Precision / Recall / F1 84.4 / 81.9 / 83.1
Pedestrian voxels (vs LiDAR body reference, 400 frames, 790 instances) Precision / Recall / F1 95.5 / 89.9 / 92.5
Detection QA (manual, 300 frames) crowded Recall / Precision / visible-miss rate 98.8 / 99.0 / 1.2
Detection QA uncrowded Recall / Precision / visible-miss rate 99.5 / 100 / 0.5
Detection QA nighttime Recall / Precision / visible-miss rate 76.9 / 97.6 / 23.1

Key Findings

  • Existing occupancy models do not work well on this dataset. The three baselines (including the adapted stereo one) reach only 24.89-27.18 mIoU and only 31.9-32.8 IoU on pedestrians, far from deployable; car (1.39-5.36) and truck (5.23-7.35) are especially low, and the authors attribute this squarely to scene distribution - vehicles are rare on a campus, whereas these classes score much higher on driving benchmarks, so the comparison must not be read across datasets. What the numbers reflect is the difficulty of the new setting: near-field, strongly imbalanced, front-view only. The authors conjecture that FlashOcc's weaker monocular pedestrian performance comes from detection errors inherited from BEVDet4D pretraining, while VoxFormer overtakes it on pedestrians (32.79 versus 31.91 / 32.45) thanks to stereo depth.
  • A panoptic head localizes pedestrians better than a pure detection head. BEVDet4D only produces pedestrian detections and reaches 41.7 AP\(_{\text{Ped}}\); Panoptic-FlashOcc with voxel-level panoptic labels raises this to 45.5, at the cost of a small mIoU drop relative to FlashOcc (27.18 to 26.64), a direct instance of the multi-task trade-off.
  • Velocity prediction is already usable but directions still fail. PF-vel reaches 0.97 m/s AVE-T over all ground-truth pedestrian voxels, slightly better than BEVDet4D's 1.00, yet on true-positive detections within 1 m BEVDet4D is better instead (AVE-D 0.36 versus 0.39) - the two trade off, and neither wins across the board. The typical qualitative failure is confusion between forward and backward walking, which indicates clear headroom in jointly predicting pedestrian velocity and panoptic occupancy.
  • Adding LiDAR is the main driver of mesh accuracy, and the gain grows monotonically with point density. On 3DPW with simulated Ouster-32 / 64 / 128 sweeps, PVE drops from 73.2 to 57.2 and then to 50.5, and MPJPE from 57.0 to 43.9 to 39.1, improving consistently across all three metrics while never using root alignment - so the gain comes from absolute placement, not only from pose.
  • At low point density, pose accuracy is not the strong suit (an easy detail to miss). In Table 3 the Ouster-32 PA-MPJPE of 47.7 is clearly worse than WHAM-B's 37.2, PLIKS's 38.5 and even CLIFF's 43.0; at Ouster-64 the 38.5 merely catches up, and only Ouster-128's 35.1 genuinely pulls ahead. In other words, the advantage under sparse point clouds shows up mainly in absolute placement (PVE / MPJPE), while articulated pose gains only materialize once points are dense enough.
  • On real LiDAR, being training-free buys cross-dataset robustness but not top scores everywhere. On SLOPER4D, Ours reaches PVE 64.4 / MPJPE 55.8, better than the image baseline CLIFF (93.7 / 84.7) but behind the LiDAR-trained LiDAR-HMR (51.9 / 51.0); on HumanM3, MPJPE 83.9 likewise trails LiDAR-HMR (77.6) and PRN (82.2). What isolates the contribution cleanly is PA-MPJPE: 43.5 versus CLIFF's 69.3 on SLOPER4D and 58.1 versus 66.5 on HumanM3, confirming that the pose improvement over the image initialization comes from the LiDAR constraints. A small counterexample is MPERE: Ours at 0.060 beats LiDAR-HMR's 0.094 but is slightly worse than LiDARCap's 0.050 and even slightly worse than its own CLIFF initialization at 0.057 - point-cloud fitting buys placement and pose while mildly distorting edge-length ratios.
  • The annotation quality itself holds up under a stricter test. Static voxels cover 96.8% of CODa's ground-truth points, meaning the voxel grid captures nearly all annotated geometry; the main source of error in the 71.2 mIoU is taxonomy and domain mismatch - CODa's "Short Vegetation" is read as terrain or vegetation depending on context under a Cityscapes taxonomy, which directly suppresses vegetation and terrain IoU. On the pedestrian side, the 84.4 / 81.9 from the bounding-box proxy is in fact pessimistic (a box counts the air around the body and the road surface at the feet as occupied, so the evaluation only keeps camera-visible occupied voxels and removes the bottom 0.1 m of each box); against a tighter reference built by voxelizing raw LiDAR returns inside each human box, precision is 95.5 and F1 is 92.5, with per-sequence F1 of 89.0 / 93.0 / 93.3 / 94.7 across sequences 7/11/16/18 - reasonably stable. One physical reason must be stated: LiDAR returns lie on the visible body surface while SMPL voxelization fills the body volume, so the two naturally differ by roughly half a body depth (about 0.19 m measured), which is why evaluation uses the standard ±1 voxel-size tolerance.
  • Nighttime is the clear weak spot, and the errors are misses rather than false positives. Daytime detection reaches 98.8 / 99.5 recall and 99.0 / 100 precision in crowded and uncrowded scenes; at night recall falls to 76.9 while precision stays at 97.6, with a 23.1% visible-miss rate - so nighttime errors are dominated by missed detections. Fortunately those misses only affect the frame they occur in and do not propagate through the static map (pedestrians were already suppressed during static mapping, and human occupancy takes precedence during alignment). The authors accordingly keep daytime and nighttime sequences separate so users can select the subset matching their deployment conditions.

Highlights & Insights

  • "Images for pose, LiDAR for placement" is realized as a three-stage procedure that requires no training. The order visibility filter, then ICP coarse alignment, then joint optimization is not decoration: first determine which parts have evidence (otherwise noise gets fitted as signal), then solve rigid placement (what ICP is good at, undisturbed by joint angles), and only then free pose and shape. This "solve by increasing degrees of freedom" order transfers to any fitting problem that must combine sparse 3D observations with 2D priors.
  • The occlusion-aware pose consistency term is a cheap and effective patch. Occluded joints have no observational evidence and are the first to drift; rather than writing an expensive interpenetration term, the paper regularizes those joints back toward their initialization with a quaternion penalty - and the authors explain the reasoning well: because occupancy is voxelized from the mesh surface, interpenetration only creates internal self-overlap and does not move the occupied/free boundary. That instinct - decide whether an error even enters the quantity you will evaluate before modeling it - is more transferable than the method itself.
  • Static/dynamic separation plus dynamic-over-static arbitration turns an annotation error into a local error. Pedestrians are first removed from static mapping with a dual (LiDAR and image) mask, and human occupancy then overrides the background during alignment. This ordering makes the global failure mode of static-map contamination impossible and degrades a missed detection into a single-frame error. Any system that builds a temporal map and overlays dynamic objects can adopt this priority scheme directly.
  • The LiDAR side of the dual mask is what makes the scheme work. With image detection alone for masking, pedestrians outside the frustum or behind occluders would still be stamped into the background; adding a per-sweep LiDAR 3D detector specifically to cover the out-of-frustum blind spot is the precondition. This "whoever can see it fills the gap" complementarity is broadly applicable.
  • Replacing bounding boxes with a body-surface reference for occupancy evaluation is a reusable improvement in evaluation methodology. Bounding-box evaluation is inherently pessimistic because of volumetric inflation; the alternative here - voxelizing raw LiDAR returns inside each human box as a sensor-facing body-surface sample and evaluating at ±1 voxel tolerance - sidesteps the absence of dense human geometry ground truth while still measuring pedestrian voxel quality up to 92.5 F1. Any work lacking dense ground truth but needing fine-grained occupancy evaluation can borrow this.
  • Hanging a velocity head on the panoptic occupancy head yields three outputs in one forward pass. Panoptic-FlashOcc-vel just adds two channels and L1 supervision to the detection head, obtaining voxel semantics, instance assignment and pedestrian velocity together, avoiding the deployment cost of one model for occupancy plus another for velocity. On compute-constrained robot platforms, this "add to an existing head" retrofit is far more realistic than standing up a separate network.

Limitations & Future Work

  • The authors acknowledge two limitations: the dataset currently covers outdoor scenes only, so cross-domain generalization (indoors, adverse weather) is uncertain; and the non-rigid model is SMPL-based and does not include carried or accompanying objects (backpacks, carts, leashed items), which underestimates the actual occupied volume of a pedestrian. They plan to extend the benchmark toward human-object occupancy.
  • More implicit boundaries are visible in the numbers. A nighttime recall of 76.9% means the pedestrian voxel annotations on the nighttime subset are themselves systematically incomplete, yet the label-quality evaluation (400 frames) is not reported by lighting condition, so "how accurate the labels are at night" has in fact never been verified - the current evidence shows that detection misses happen, not that the resulting labels stay precise. Supporting nighttime deployment would require an audit of label quality stratified by illumination.
  • The evaluation scope is narrow in another way: CODa annotates point-level semantics only for a small portion of its sequences, so static label quality is reported only for road, vegetation and terrain, leaving classes such as pole and two-wheeler - equally important in crowd scenes - without ground truth. The tighter pedestrian reference is still a sampling of the visible body surface, so whether the mesh's far side and interior fill are correct cannot be verified by bounding boxes or LiDAR returns at all; this is an inherent blind spot at the dataset level.
  • One concrete improvement: the reported MPERE suggests LiDAR fitting mildly distorts edge-length ratios (0.060, slightly worse than the 0.057 of the image initialization), so edge-length or bone-length consistency could be added to \(\mathcal{L}_{\text{total}}\) as an explicit regularizer to protect metric consistency while gaining placement accuracy.
  • Another direction worth pursuing is moving the benchmark from perception metrics to navigational usefulness: the limb-level geometry preserved at 0.02 m should be used to check whether narrow gaps in crowds are correctly judged traversable and whether trajectories under social-distance constraints are safe, but the downstream validation here stops at two perception tasks, with no closed-loop evidence from the planning side.
  • vs Occ3D / OpenOccupancy / SSCBench (driving occupancy datasets): they build on nuScenes / Waymo / KITTI-360 with large scale and dense voxel labels, but dynamic objects (mostly vehicles) are handled as rigid bodies with bounding boxes, pedestrians are rare and rigidified, and instance tracking is absent. MobileOcc supplies a completely different piece: pedestrian-dense, near-field, human-centric annotation that expresses non-rigid shape with SMPL meshes and provides per-instance IDs. The cost is far smaller scale and class coverage than driving datasets - the two are complementary rather than substitutable.
  • vs SemanticKITTI / OpenScene: SemanticKITTI does not account for moving objects and has limited scale and diversity; OpenScene provides instance tracking but still within a rigid modeling paradigm. MobileOcc is distinguished by having instance tracking and non-rigid human meshes at the same time (exactly the conclusion of Table 1 in the paper).
  • vs WildOcc: WildOcc extends RELLIS-3D into dense semantic occupancy for off-road scenes and targets unstructured environments; MobileOcc targets humans. Both sit outside the structured-road setting, but the modeled objects are orthogonal.
  • vs image/video-only HMR (CLIFF, ROMP, HybrIK, WHAM, MotionBERT): they infer 3D from 2D alone and mitigate depth and scale ambiguity with post-hoc root alignment; this paper treats such a prediction as an initialization rather than an answer and pulls it to the true position in the sensor frame with LiDAR, which yields the PVE / MPJPE advantage at the cost of dependence on point-cloud quality (at low density, PA-MPJPE is actually worse than for image-only methods).
  • vs LiDAR-only human fitting (LiDARCap, LiDAR-HMR, VoteHMR, SAHSR, S3): they rely on LiDAR-specific (and often dataset-specific) training or use implicit fields, and become brittle when human points overlap nearby objects or thin out at distance. This work is training-free and runs directly across datasets, accepting that it is not first on every metric (PVE / MPJPE on SLOPER4D still trail LiDAR-HMR) in exchange for generalization and lower deployment cost.
  • vs FlashOcc / Panoptic-FlashOcc / VoxFormer (occupancy methods): this paper does not alter their architectures; it measures their shortcomings in a new setting (front-view, near-field, crowded) and adds a velocity branch to the panoptic head along the way. For occupancy researchers, MobileOcc offers a place where driving settings fail to expose problems: monocular pedestrian detection errors inherited from pretraining, nighttime misses, and direction confusion.

Rating

  • Novelty: ⭐⭐⭐⭐ First semantic occupancy dataset for mobile robots with non-rigid human modeling and instance tracking; replacing rigid bounding boxes with an image-LiDAR fusion mesh optimization that needs no training, delivered together with the evaluation protocol - the gap is identified precisely.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Label quality is assessed against two references (bounding boxes and LiDAR body surface) plus stratified detection robustness plus multi-baseline evaluation on two tasks; but static labels have ground truth for only three classes, nighttime label quality is not audited separately, and there is no planning-side validation.
  • Writing Quality: ⭐⭐⭐⭐ The role of and rationale for each pipeline stage are clear, trade-offs such as dropping the interpenetration term are argued concretely, and metric definitions are complete; some equations are corrupted in typesetting/extraction, and a few numbers require checking against the original tables.
  • Value: ⭐⭐⭐⭐ A plug-and-play testbed for teams working on crowd navigation and near-field perception; the static/dynamic separation with dynamic-over-static arbitration and the body-surface evaluation idea both transfer, while the downstream value for crowd interaction and safe navigation remains to be realized by follow-up work.