Skip to content

title: >- [Paper Note] TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception description: >- [ECCV 2026][Autonomous Driving][Occupancy Perception] Camera-LiDAR-RADAR 4D occupancy world model learning via continuous velocity self-supervision and geometry pseudo-labeling. tags: - ECCV 2026 - Autonomous Driving - Occupancy World Model - Self-Supervised Learning - Radar-LiDAR-Camera Fusion date: 2026-09-19 content_hash: 9b01c0299003defe

TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Conference: ECCV 2026
Paper: ECCV 2026
Area: Autonomous Driving
Keywords: 4D occupancy grid, multimodal fusion, unsupervised world model, obstacle segmentation, anything perception

TL;DR

TriO establishes an unsupervised 4D continuous occupancy world model leveraging camera, LiDAR, and RADAR as both inputs and self-supervisory sources, achieving robust zero-shot road obstacle segmentation and flow forecasting without human annotations.

Background & Motivation

For an autonomous driving vehicle to navigate safely through intricate traffic environments, it must be able to perceive anything—reliably distinguishing any path-blocking obstacle from traversable drivable surfaces—and subsequently forecast the continuous evolution of the scene to guide downstream motion planning. Classical perception and prediction pipelines rely on 3D object detection bounding boxes coupled with trajectory prediction, which are bound to thresholding detections and a restricted set of known object classes. While dense 3D and 4D occupancy grid models circumvent detection thresholding by estimating spatial occupancy fields directly, existing supervised occupancy formulations depend on dense voxel-level human annotations, incurring prohibitive costs and remaining confined to a closed vocabulary that fails to capture safety-critical rare edge cases.

To alleviate annotation bottlenecks, recent self-supervised occupancy paradigms exploit LiDAR ray-casting projections or distill 2D open-vocabulary semantic pseudo-labels from vision-language foundation models. However, pure LiDAR self-supervision suffers from physical degradations such as beam blooming, multipath returns, and sparse range attenuation. On the other hand, distilling pseudo-labels from text-prompted vision-language models (e.g., GroundedSAM, OWLv2, SAM) exhibits severe failure modes when encountering tail-distribution road obstacles like fallen cones, lost cargo, or small animals, even when provided with oracle ground-truth textual descriptions. Furthermore, automotive RADAR—which uniquely offers weather-resilient long-range sensing and direct instantaneous radial velocity via the Doppler effect—remains fundamentally overlooked in self-supervised 4D multimodal world models.

To resolve these tensions, this paper bypasses brittle text-prompted semantics and grounds scene understanding in fundamental geometric traversability. Core idea: integrate camera, LiDAR, and RADAR into a unified self-supervised continuous 4D world model, where monocular surface normals and LiDAR planar patches generate geometry-driven continuous soft obstacle labels, while Doppler RADAR measurements self-supervise continuous 3D velocity flow fields without manual annotations.

Method

Overall Architecture

TriO adopts an encoder-implicit decoder architecture operating across continuous space-time coordinates \((x, y, z, t)\) to predict spatial occupancy probability \(o \in [0, 1]\), path-blocking obstacle segmentation probability \(s \in [0, 1]\), and continuous 3D flow vector \(\mathbf{f} \in \mathbb{R}^3\). The network ingests historical sensor observations comprising the most recent monocular RGB image frame, multi-frame LiDAR point sweeps, and multi-frame RADAR target sweeps.

The overall pipeline executes four interconnected stages: multimodal decoupled feature encoding, cross-modal voxel-attention fusion between sparse 3D point clouds and 2D vision representations, multiscale spatial densification with deformable attention, and implicit query decoding. In parallel, self-supervised targets are generated online directly from physical observations: LiDAR ray-casting for occupancy, geometry-guided confidence fusion for obstacle segmentation, and Doppler projection for scene flow.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multimodal Spatiotemporal Input<br/>Monocular Image + Multi-frame LiDAR + RADAR"] --> Enc["Decoupled Multimodal Encoding<br/>ResNet-50 Vision + Sparse Voxel MLP Encoders"]
    Enc --> VoxelAtt["Cross-Modal Voxel-Attention Fusion<br/>Sparse 3D Voxel Query on 2D Image Tokens"]
    VoxelAtt --> DefAtt["Deformable Attention Densification<br/>Multiscale Feature Receptive Field Expansion"]
    DefAtt --> Dec["Implicit Continuous 4D Decoder<br/>Spatiotemporal Coordinate Query for o, s, f"]
    Dec --> Out["Continuous 4D Predictions<br/>Occupancy o + Obstacle Probability s + 3D Flow f"]

Key Designs

1. Decoupled multimodal encoding and cross-modal voxel-attention fusion: bridging high-resolution appearance and sparse spatiotemporal dynamics Prior sensor fusion methods frequently perform naive concatenation or shallow projection, which fails to align dense visual textures with sparse spatiotemporal point clouds. TriO employs separate specialized encoders tailored to sensor physics: the vision branch encodes the single most recent camera frame using a pretrained ResNet-50 backbone to produce multiscale 2D features \(Z_c\), while LiDAR and RADAR measurements are vectorized with per-point physical attributes. Specifically, LiDAR points include intensity \(i\) and relative timestamp \(\Delta t\) (\(l = [x, y, z, i, \Delta t]^\top\)), and RADAR targets incorporate range rate \(v_r\), radar cross-section \(\sigma\), and timestamp \(\Delta t\) (\(r = [x, y, z, v_r, \sigma, \Delta t]^\top\)). These points are embedded via lightweight MLPs and mapped into a shared 3D sparse feature volume \(Z_{lr}\). At the coarsest voxel scale, sparse 3D voxels cross-attend to 2D image features through a voxel-attention mechanism, generating multiscale volumes \([V_{2\times}, V_{4\times}, V_{8\times}, M_{16\times}]\). The coarsest map \(M_{16\times}\) is densified and passed through deformable attention layers to expand spatial context before feeding the query decoders.

2. Geometry-driven image-LiDAR soft pseudo-label generation: prompt-free road obstacle segmentation Open-set segmentors often miss unusual or out-of-vocabulary road debris due to semantic distribution shifts. TriO addresses this by deriving obstacle supervision purely from geometric consistency relative to the ground surface. For camera imagery, the Metric3Dv2 foundation model predicts dense metric depth, surface normals, and corresponding pixel confidences, yielding continuous image obstacle probabilities \(p_{\text{obst}}^{\text{image}}\) and validity weights \(p_{\text{valid}}^{\text{image}}\). For LiDAR point sweeps, Patchwork++ partitions the horizontal plane into polar patches, fitting local ground plane centroids and surface normals to compute point-to-plane elevation distances, mapped into probabilities \(p_{\text{obst}}^{\text{LiDAR}}\) and validity \(p_{\text{valid}}^{\text{LiDAR}}\). A confidence-arbitrated fusion rule combines both sources:

\[ \mathbf{p}^{\text{comb}} = \begin{cases} \mathbf{p}^{\text{image}}, & \text{if } (p_{\text{valid}}^{\text{LiDAR}} < 0.5) \wedge (p_{\text{valid}}^{\text{image}} > p_{\text{valid}}^{\text{LiDAR}}) \\ \mathbf{p}^{\text{LiDAR}}, & \text{otherwise} \end{cases} \]

This strategy leverages LiDAR's metric precision in open free spaces while allowing high-resolution camera normals to resolve low-profile, close-to-ground obstacles, producing continuous soft supervisory targets \(\mathbf{p}^{\text{comb}} = [p_{\text{valid}}^{\text{comb}}, p_{\text{obst}}^{\text{comb}}]^\top\).

3. Doppler-guided RADAR flow self-supervision: annotation-free continuous 3D velocity learning Conventional unsupervised scene flow relies on iterative closest point or cross-frame correspondence, which degrade severely at long ranges or over sparse object returns. TriO capitalizes on radar Doppler physics as direct velocity supervision. Measurements with high uncertainty are first filtered out to yield a clean target set \(\mathcal{R}_{\text{clean}}\). For any query coordinate \(\mathbf{q}\), the predicted absolute 3D velocity vector \(\mathbf{f}_{\theta, \text{flow}}(\mathbf{q})\) is transformed into the sensor frame of reference by subtracting the ego-vehicle velocity \(\mathbf{v}_{\text{sensor}}\). The resulting relative flow is projected onto the unit radial line-of-sight vector \(\mathbf{d}_{\text{radar}}\):

\[ f_{\text{rad}}(\mathbf{q}) = \mathbf{d}_{\text{radar}} \cdot \left( \mathbf{f}_{\theta, \text{flow}}(\mathbf{q}) - \mathbf{v}_{\text{sensor}} \right) \]

The scalar projected velocity is then directly aligned with the raw radar Doppler reading \(v_{\text{doppler}}\), supervising dynamic scene flow purely through physical geometric projection.

Loss & Training

The entire network is trained end-to-end using a unified multitask loss function:

\[ \mathcal{L} = \lambda_1 \mathcal{L}_{\text{occ}} + \lambda_2 \mathcal{L}_{\text{flow}} + \lambda_3 \mathcal{L}_{\text{seg}} \]
  1. Occupancy Loss \(\mathcal{L}_{\text{occ}}\): Computed via binary cross-entropy (BCE) over LiDAR ray-cast free-space points \(\mathcal{O}^-\) and occupied surface-back points \(\mathcal{O}^+\).
  2. Obstacle Segmentation Loss \(\mathcal{L}_{\text{seg}}\): Formulated as a Soft Focal Loss over continuous soft targets \(p_{\text{obst}}^{\text{comb}}\), scaled dynamically by label validity confidence \(p_{\text{valid}}^{\text{comb}}\):
\[ \mathcal{L}_{\text{seg}} = \frac{1}{|\mathcal{Y}_{\text{seg}}|} \sum_{(\mathbf{q}, \mathbf{p}^{\text{comb}}) \in \mathcal{Y}_{\text{seg}}} \text{SFocalLoss}\left(f_{\theta, \text{seg}}(\mathbf{q}),\; p_{\text{obst}}^{\text{comb}}(\mathbf{q}),\; p_{\text{valid}}^{\text{comb}}(\mathbf{q})\right) \]
  1. Flow Loss \(\mathcal{L}_{\text{flow}}\): Computed as an \(L_1\) regression loss over the cleaned RADAR target set \(\mathcal{R}_{\text{clean}}\) comparing projected radial flow against Doppler velocity:
\[ \mathcal{L}_{\text{flow}} = \frac{1}{|\mathcal{R}_{\text{clean}}|} \sum_{\mathbf{q} \in \mathcal{R}_{\text{clean}}} \left| f_{\text{rad}}(\mathbf{q}) - v_{\text{doppler}} \right| \]

Key Experimental Results

Main Results

The method is evaluated across three real-world autonomous driving benchmarks: zero-shot anomaly obstacle segmentation on Spotting the Unexpected (STU, 22 labeled sequences, trained on AV2 and evaluated without fine-tuning), point cloud forecasting on the Argoverse 2 (AV2) Occupancy Forecasting leaderboard, and downstream fine-tuning on the 350-meter Long Range RADAR (LRR) dataset.

Table 1: Semantic occupancy prediction performance on STU (zero-shot transfer from AV2)

Method / Supervision Source Supervision Type mIoU ↑ AP ↑ F1 ↑ Recall ↑
UnO (CSL) LiDAR-only self-supervision 40.6 23.1 29.8 25.5
DiO (CSL) LiDAR-only self-supervision 49.8 74.8 72.9 65.6
TriO (MCPP + SAM) Open-vocabulary text distillation 27.9 25.7 27.5 24.0
TriO (GroundedSAM) Open-vocabulary text distillation 50.4 50.1 52.1 54.8
TriO (OWLv2 + SAM) Open-vocabulary text distillation 50.1 72.3 70.1 68.8
TriO (Patchwork++) LiDAR geometric distillation 39.4 74.8 70.8 59.5
TriO (ISL) (Ours) Monocular image geometric soft labels 41.8 57.1 53.7 59.9
TriO (CSL Hard) (Ours) Discretized multimodal geometric labels 50.3 76.1 71.7 64.5
TriO (CSL Soft) (Ours) Continuous multimodal soft labels 54.9 81.6 77.7 71.3

Table 2: LiDAR point cloud forecasting on the Argoverse 2 leaderboard

Method L1 (m) ↓ AbsRel (%) ↓ NFCD (\(m^2\)) ↓ CD (\(m^2\)) ↓
RayTracing 4.88 35.00 3.62 17.03
4D-Occ 3.22 19.00 2.45 72.74
Progressive ARM 2.32 13.00 1.81 71.41
UnO 2.24 12.00 0.86 8.10
DIO 2.11 11.00 0.85 13.96
TriO (Ours) 1.82 10.63 0.93 10.90

Table 3: Downstream occupancy and flow forecasting fine-tuning on LRR (All vs. 200m+ range)

Method All mAP ↑ All Soft-IoU ↑ All EPE ↓ 200m+ mAP ↑ 200m+ Soft-IoU ↑ 200m+ EPE ↓
Implicit-O 49.1 22.6 5.69 29.5 10.7 10.2
UnO 51.6 21.2 4.92 27.2 8.30 8.29
DIO 58.0 31.5 3.01 36.2 13.8 4.40
TriO (Ours) 60.5 35.7 2.86 42.6 21.4 4.01

Ablation Study

Table 4: Ablation analysis of TriO components and sensor modalities on LRR

Configuration All mAP ↑ All Soft-IoU ↑ All EPE ↓ 200m+ mAP ↑ 200m+ Soft-IoU ↑ 200m+ EPE ↓ Note
TriO (full model) 60.5 35.7 2.86 42.6 21.4 4.01 Full trimodal self-supervised pipeline
w/o pretraining 59.7 33.9 4.09 41.0 19.8 4.70 End-point error degrades by 1.23 overall
w/o radar input 57.7 31.9 3.04 34.5 14.3 4.59 200m+ mAP plummets by 8.1, demonstrating radar necessity
w/o camera input 60.5 35.7 2.68 42.2 20.9 3.64 Single-frame camera adds slight flow noise while assisting far geometry
w/o radar loss 60.5 35.6 4.35 42.6 21.3 6.02 EPE degrades from 4.01 to 6.02 at 200m+, confirming Doppler loss

Key Findings

  • Geometric soft labels outperform open-vocabulary text VLMs: On the zero-shot STU benchmark, distilling CSL Soft into TriO yields 54.9 mIoU and 81.6 AP, outperforming state-of-the-art vision-language distillation (OWLv2 + SAM achieves 50.1 mIoU and 72.3 AP). Geometric surface continuity and elevation are significantly more dependable than text embeddings for detecting rare, unmodeled hazards.
  • RADAR is indispensable for extreme-range perception: Removing RADAR input degrades 200m+ occupancy mAP from 42.6% down to 34.5%, and Soft-IoU from 21.4% to 14.3%. Automotive radar provides vital signal returns at long distances where optical sensors attenuate.
  • \(100\times\) label sample efficiency: In downstream supervised fine-tuning, TriO pretrained with unsupervised multimodal signals reaches competitive mAP using only \(10^2\) labeled frames, matching the performance of baseline models trained from scratch that require \(10^4\) fully labeled frames.

Highlights & Insights

  • Closed-loop multimodal self-supervision: By repurposing camera normal priors, LiDAR beam geometry, and RADAR Doppler velocities as simultaneous inputs and supervisory signals, TriO eliminates dependencies on dense 3D manual labels or bounding box annotations.
  • Continuous soft probability calibration: Replacing discretized binary thresholds with continuous soft targets and a Soft Focal Loss allows the implicit decoder to model geometric ambiguity near ground boundaries, markedly improving calibration quality and average precision.
  • Physics-grounded velocity projection: Mapping 3D ego-compensated velocity vectors directly onto RADAR line-of-sight unit vectors provides a clean, elegant mechanism for training dynamic 3D flow without temporal correspondence searching.

Limitations & Future Work

  • RADAR occupancy supervision remains unaddressed: RADAR returns are currently leveraged solely for velocity flow supervision; future work could formulate continuous inverse sensor models to extract occupancy supervision from radar echoes despite multipath noise and clutter.
  • Lack of temporal multi-view vision: The visual backbone currently processes only the most recent camera frame; extending visual encoding across historical multi-camera sequences could eliminate transient flow noise and strengthen spatiotemporal tracking.
  • Limited scale of anomaly benchmarks: Existing zero-shot benchmarks evaluating long-tail obstacle segmentation (e.g., STU) comprise relatively few test sequences; broader public benchmarks under adverse weather conditions are needed to further stress-test anything perception.
  • vs UnO (Agro et al., CVPR 2024) & DIO (Diehl et al., CVPR 2025): While UnO and DIO introduced self-supervised continuous implicit 4D occupancy-flow modeling using LiDAR ray tracing, they lack camera context and RADAR Doppler guidance. TriO expands to trimodal operation and introduces explicit zero-shot obstacle segmentation capabilities.
  • vs 4D-Occ (Khurana et al., CVPR 2023): 4D-Occ uses explicit voxel grids for point cloud forecasting, which suffer from memory bottlenecks and constrained spatial resolution. TriO utilizes an implicit continuous coordinate decoder with flexible query resolution and lower compute overhead.
  • vs Open-Vocabulary 2D Segmentors (GroundedSAM, OWLv2, SAM): Vision-language models fail on long-tail driving hazards due to vocabulary misalignment and detection misses. TriO formulates anything perception as geometric road-obstacle separation, exhibiting superior zero-shot transferability.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant integration of camera, LiDAR, and RADAR into a mutually supervisory 4D world model, uniquely combining Doppler physics and planar patch geometry.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning AV2 point cloud forecasting, STU zero-shot anomaly transfer, 350m LRR long-range benchmarks, and thorough ablation studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear problem motivation, sound mathematical formulations, and insightful articulation of why pure geometry surpasses text-prompted VLMs in safety-critical edge cases.
  • Value: ⭐⭐⭐⭐⭐ Provides an extensible, annotation-free foundation for robust 4D perception, long-range world models, and obstacle avoidance in autonomous driving.