HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training¶

Conference: ECCV 2026
arXiv: 2606.20189
Code: https://maxiuw.github.io/hilda
Area: Autonomous Driving / Self-Supervised Learning / Cross-Modal Knowledge Distillation
Keywords: Autonomous Driving, LiDAR Pre-training, Cross-Modal Distillation, Vision Foundation Models, Occupancy Diffusion

TL;DR¶

HilDA distills LiDAR backbones using the "hierarchical semantic structure + CLS global context" from Vision Foundation Models (VFMs) rather than just the final layer features. It additionally incorporates a conditional diffusion task predicting future BEV occupancy to complement spatio-temporal geometric information, setting a new SOTA across multiple downstream tasks in cross-modal distillation: segmentation, detection, scene flow, and semantic occupancy.

Background & Motivation¶

3D perception in autonomous driving has long suffered from annotation bottlenecks: LiDAR point clouds are geometrically accurate but semantically sparse and lack texture, while frame-by-frame manual annotation is expensive and fails to cover the combinatorial explosion of geometry and motion in the real world. In recent years, the mainstream approach has been to treat image-trained vision foundation models (such as DINOv2) as teachers, utilizing camera-LiDAR calibration correspondences to "distill" dense semantic features from 2D into 3D backbones as a form of self-supervised pre-training. Such methods (SLidR, Seal, ScaLR, CleverDistiller, LiMA, etc.) have successfully demonstrated that 3D networks can inherit semantic representations robust to domain shifts and adverse weather conditions.

However, the authors observe a common limitation: almost all existing methods only distill the output features of the final layer of the VFM, treating the teacher as a black box. This leads to two critical losses. First, ViT semantics are constructed progressively layer by layer; intermediate layers are often more useful for downstream tasks. Focusing solely on the final layer discards the hierarchical information chain of how features are gradually formed. Second, the ViT CLS token encodes the global context of the entire scene (e.g., highway vs. residential area, which objects are most salient), whereas existing distillation methods uniformly discard the CLS token, performing only point-to-pixel local alignment. Furthermore, image VFMs fundamentally lack 3D geometry and temporal cues. While some works attempt to patch this with an auxiliary "occupancy prediction" task, they typically rely on discriminative supervision like voxel-wise binary cross-entropy. This only models local marginal distributions, providing no explicit incentive for "globated consistent 3D structures" and "temporal object permanence."

Consequently, this work unifies these two perspectives. Onto the semantic side, instead of merely learning the teacher's final answer, this work advocates learning "how it thinks"—simultaneously distilling multi-layer intermediate features and the global CLS token, termed hierarchical distillation. Onto the geometric and temporal side, rather than using discriminative voxel-wise occupancy prediction, a generative diffusion model is employed to model the joint distribution of occupancy: formulating "future occupancy prediction" as conditional denoising, thereby enabling the model to internalize spatial manifolds and motion during the coarse-to-fine denoising process. Core Idea: The semantic what of the VFM is transferred into the LiDAR backbone via hierarchical distillation (multi-layer + CLS), which is then complemented by a conditional diffusion task predicting future BEV occupancy to inject the geometric where (generative spatio-temporal dynamics). The synergy of these two components yields a 3D representation proficient in both semantics and structure.

Method¶

Overall Architecture¶

HilDA is a purely self-supervised LiDAR backbone pre-training framework. The inputs consist of synchronized multi-view camera images and a LiDAR sequence (three frames \(\{t_{-1}, t_0, t_1\}\), where features are extracted only from the past and current frames \(\{t_{-1}, t_0\}\), and the future frame \(t_1\) serves solely as the target for occupancy prediction). The output is a trained 3D backbone \(S_\theta\) (MinkUnet34 by default). During inference, all auxiliary pre-training components are discarded, leaving only the 3D backbone for downstream perception tasks. The pre-training process is driven by three concurrently optimized self-supervised objectives: with a frozen VFM (DINOv2) acting as the teacher, it first performs multi-layer point-to-pixel distillation to transfer progressively constructed local semantics, followed by global context distillation that aligns the teacher's CLS token with a learnable 3D global token, and finally appends a temporal occupancy diffusion task that denoises future BEV occupancy conditioned on the backbone features of the past two frames. The first two objectives constitute "hierarchical distillation" (addressing the semantic what), while the latter delivers the geometric where.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input<br/>LiDAR Sequence + Multi-View Images"] --> B["3D Student Backbone S_θ<br/>Extract Hierarchical Features at t-1 / t0"]
    A --> T["Frozen VFM Teacher<br/>DINOv2 Multi-Layer Features + CLS"]
    B --> C["Multi-Layer Point-Pixel Distillation<br/>Separate Alignment for Final Two Layers (Cosine)"]
    T --> C
    B --> D["Global Context Distillation<br/>CLS ↔ Learnable 3D Global Token"]
    T --> D
    B --> E["Temporal Occupancy Diffusion<br/>Conditional DDPM Denoising of Future BEV Occupancy"]
    C --> F["Pre-trained Backbone S_θ<br/>Downstream: Seg / Det / Scene Flow / Occ"]
    D --> F
    E --> F

Key Designs¶

1. Multi-Layer Distillation: Learning "How to Construct Semantics" Instead of Just the Final Answer

Existing cross-modal distillation techniques typically align only the final layer of the VFM, discarding the hierarchical information chain of "progressive feature abstraction" in intermediate layers, which are often more effective for downstream tasks. HilDA projects features from multiple layers of the teacher to corresponding layers of the student (defaulting to the final two layers), encouraging the 3D student to replicate how features are constructed rather than just what features are output. Specifically, camera-to-LiDAR geometric calibration is used to project 3D points onto the pixel plane, yielding a set of valid point-pixel pairs \(\mathcal{M}_c\). For each layer, camera, and point-pixel pair, the cosine distance is minimized. The student point features are first mapped through a lightweight, layer-specific MLP \(\mathcal{H}_\ell\) to match dimensions, and then compared with the teacher's pixel features:

\[\mathcal{L}_{\text{distill}} = \frac{1}{V(K{+}1)} \sum_{\ell=L-K}^{L} \sum_{c=1}^{V} \frac{1}{|\mathcal{M}_c|} \sum_{(i,j)\in\mathcal{M}_c} \left( 1 - \frac{\mathcal{H}_\ell(\mathbf{f}_{i,\ell}) \cdot \mathbf{q}_{j,\ell,c}}{\|\mathcal{H}_\ell(\mathbf{f}_{i,\ell})\|_2 \, \|\mathbf{q}_{j,\ell,c}\|_2} \right)\]

A critical design choice is what the authors term Separate late-layer matching: only matching adjacent late layers individually (\(Q_L{\to}F_L\), \(Q_{L-1}{\to}F_{L-1}\)), rather than extending to earlier layers or concatenating multi-layer features into a single target. This is because earlier layers of MinkUnet34 aggregate coarser and more irregular 3D neighborhoods, where sparse voxels might project to semantically disparate regions, and extremely early VFM layers do not align well in abstraction level with late LiDAR features. Ablation studies (Tab. 8) demonstrate that separately matching the final two layers (53.8 LP) significantly outperforms using only the final layer (46.4) or only the penultimate layer (49.9). It also beats KR-style sliding window aggregation, concatenation aggregation, and cross-layer matching—proving that the gains stem from "preserving correspondences at compatible late-stage abstraction levels" rather than simply stacking more teacher features. Using cosine distance instead of \(\ell_2\) or KL divergence is also crucial: since the ViT teacher and Minkowski student differ in architectures and feature norm distributions, cosine distance supervises only the semantic direction without imposing cross-architecture scale matching.

2. Global Context Distillation: Mapping the Teacher's CLS Token to a Learnable 3D Global Token

Point-to-pixel local distillation excels at aligning fine-grained semantics but fails to capture the overall context of the entire scene (e.g., distinguishing highway environments from residential areas). ViT compresses this global information into its CLS token, which existing methods universally discard. HilDA fills this gap: on the teacher side, the CLS tokens from the final layer of all camera images are max-pooled to construct a unified visual scene descriptor \(\text{CLS}_{Q_L}\); on the student side, a dedicated 3D global "token" is designed—the point features from the final layer of the backbone are mapped through a dedicated MLP projection head \(\mathcal{H}_{\text{CLS}}\), followed by global max-pooling over all points in the scene to obtain \(\text{CLS}_{F_L}\). The student token is then aligned with the teacher descriptor using Mean Squared Error (MSE):

\[\mathcal{L}_{\text{cls}} = \| \text{CLS}_{F_L} - \text{CLS}_{Q_L} \|_2^2\]

This loss acts as a scene-level regularizer, urging the 3D backbone to aggregate global contextual clues, whereas max-pooling highlights the most active responses and scales up salient view-level and point-level activations. Ablations (Tab. 9) reveal that max-pooling outperforms learnable pooling and view/frustum-wise student CLS—proving that global alignment benefits from "emerging highly salient responses." Furthermore, this formulation is calibration-free (independent of point-to-pixel projections), making it insensitive to camera-LiDAR misalignments, which nicely compensates for the local distillation's sensitivity to calibration errors.

3. Temporal Occupancy Diffusion: Formulating "Future Occupancy Prediction" as Conditional Denoising to Inject Generative Spatio-Temporal and Motion Priors

While distillation infuses semantic features, it lacks explicit modeling of spatio-temporal dynamics. The authors introduce a label-free auxiliary task: predicting future BEV occupancy using conditional diffusion, forcing the backbone to learn predictive spatial representations. Given past frames \(P_{t-1}, P_{t_0}\), \(S_\theta\) encodes them into features and collapses them into a shared dense BEV history feature map \(C_{\text{history}}\). The future frame \(P_{t_1}\) is transformed into the \(t_0\) coordinate system, ground-filtered, and projected into a binary BEV occupancy target \(x_{\text{occ}}\). Future occupancy prediction is formulated as conditional DDPM denoising: at diffusion step \(\tau\), the target is perturbed to \(x_\tau = \sqrt{\bar\alpha_\tau}\,x_{\text{occ}} + \sqrt{1-\bar\alpha_\tau}\,\epsilon\), and a 2D UNet denoising network predicts the noise \(\epsilon_\theta(x_\tau, \tau, C_{\text{history}})\) conditioned on the history features (via channel concatenation), trained with a hybrid objective:

\[\mathcal{L}_{\text{diffusion}} = \underbrace{\mathbb{E}_{\tau,\epsilon,C_{\text{history}}}\left[ \| \epsilon - \epsilon_\theta(x_\tau, \tau, C_{\text{history}}) \|_2^2 \right]}_{\text{Noise Prediction}} + \lambda \underbrace{\| x_{\text{occ}} - \hat{x}_{\text{occ}} \|_2^2}_{\text{Occupancy Reconstruction}}\]

The first term supervises noise prediction, and the second term (weighted by \(\lambda\)) provides a complementary reconstruction signal to refine the conditioning at lower noise levels. Why use diffusion instead of a conventional occupancy decoder? Because discriminative voxel-wise occupancy prediction only models local marginal distributions, offering no incentive for "globally consistent 3D configurations." Occupancy, by nature, exhibits spatio-temporally correlated continuous structures that respect object permanence. Diffusion joint-models the occupancy distribution, enforcing coarse-to-fine (global structures first, then local details) scene-level spatio-temporal constraints through iterative multi-step denoising. Direct comparisons in the ablation study (Tab. 9) show that replacing diffusion with a simple decoder (49.1) or ALSO (54.7) performs worse than diffusion (56.3). Moreover, this task brings the largest occupancy gain to dynamic and object classes, as future occupancy directly supervises object permanence and short-term motion.

Loss & Training¶

HilDA is optimized end-to-end with a weighted sum of the three objectives:

\[\mathcal{L}_{\text{total}} = \omega_{ds}\mathcal{L}_{\text{distill}} + \omega_{gl}\mathcal{L}_{\text{cls}} + \omega_{df}\mathcal{L}_{\text{diffusion}}\]

The teacher uses DINOv2 (ViT-S/B/L) and the student uses MinkUnet34 (PTv3 is also supported). Pre-training is performed only once on the nuScenes training set using synchronized RGB-LiDAR and calibration data, without using any task annotations. The pre-trained backbone is directly transferred to all downstream benchmarks without re-pretraining on target datasets. In the paper, HilDA† denotes the variant without the auxiliary diffusion loss, highlighting the independent contribution of the diffusion task (excluded during rankings, comparing only standard HilDA against previous SOTA).

Key Experimental Results¶

Main Results¶

Main results of cross-modal distillation for 3D semantic segmentation (pre-trained on nuScenes, evaluated on nuScenes / SemanticKITTI / Waymo; MinkUnet34 backbone, LP = Linear Probing). The gains are most significant in data-sparse scenarios (1%–10%):

Scenario (ViT-B Teacher)	Metric	HilDA	Prev. SOTA	Gain
nuScenes LP	mIoU	58.95	56.65 (LiMA)	+2.30
nuScenes 1%	mIoU	62.71	59.80 (CleverDistiller)	+2.91
nuScenes 5%	mIoU	70.19	66.44 (CleverDistiller)	+3.75
SemanticKITTI 1%	mIoU	53.44	51.48 (CleverDistiller)	+1.96
Waymo 1%	mIoU	53.89	53.56 (CleverDistiller)	+0.33

Multi-task transfer performance (on the same ViT-B/MinkUnet34 backbone, with frozen or re-initialized heads) is also comprehensively superior:

Task	Dataset	Metric	HilDA	Previous Best
3D Detection	KITTI 20%	mAP	71.0	67.1 (CD)
3D Detection	nuScenes 20%	mAP	57.9	55.1 (ScaLR)
3D Semantic Occupancy	nuScenes	mIoU	20.0	16.5 (CD)
Scene Flow	Argoverse 2	3-way EPE Mean↓	0.021	0.024 (CD)
Robustness	nuScenes-C (LP)	mCE↓ / mRR↑	124.27 / 88.20	137.23 / 79.76

Regarding robustness, HilDA shows particularly large improvements under heavy perturbations and highly sparse corruptions, such as Cross-Sensor and Snowy conditions.

Ablation Study¶

Component analysis (Tab. 7, base = ScaLR + MLP projection head, metrics are nuScenes LP / 1%, etc.):

Configuration	Diff	Distill	CLS	nuSc LP	nuSc 1%
base	✗	✗	✗	46.36	55.01
(a) Add Diffusion	✓	✗	✗	50.43	56.97
(b) Add Multi-Layer Distill	✗	✓	✗	53.77	57.03
(c) Multi-Layer + CLS	✗	✓	✓	55.13	57.81
(d) Diffusion + Multi-Layer	✓	✓	✗	55.53	59.04
(e) Full HilDA	✓	✓	✓	56.29	59.46

Key Findings¶

Adding diffusion (a) or hierarchical distillation (c) individually significantly improves the baseline, and their combined use (e) achieves the highest scores. This implies that "discriminative semantic distillation" and "generative spatio-temporal diffusion" are complementary and synergetic rather than redundant.
The design choices for multi-layer distillation are highly sensitive: matching the final two layers "separately" yields the best performance, while extending to earlier layers drops performance. The gains stem from "preserving late-layer correspondences with compatible levels of abstraction," not simply from stacking more teacher layers.
Within the distillation losses, cosine distance (56.3) performs far better than \(\ell_2\) (53.8) and KL divergence (52.1), demonstrating that cross-architecture supervision should target semantic directions rather than exact feature scales.
The downstream gains from diffusion are primarily concentrated on dynamic/object classes (since future occupancy provides direct supervision of object permanence and short-term motion). Gains for ground classes are smaller, as the ground-filtered BEV target discards vertical cues needed to distinguish terrain from sidewalks.

Highlights & Insights¶

“Learning how the teacher thinks, not just its answer” — elevating knowledge distillation from “aligning the final layer” to “aligning the hierarchical structure + global CLS”. This viewpoint is easily transferable to any cross-modal distillation using large foundation models as teachers (not limited to LiDAR).
The coupling of discriminative (distillation) and generative (diffusion) tasks is ingenious: utilizing diffusion alone tends to over-emphasize local details and weaken semantic discriminative capability, but combining it with a strong discriminative teacher (DINOv2) preserves semantics while supplementing geometry. The two are complementary counterparts, with neither being replaceable by the other.
Global CLS distillation is calibration-free, rendering it naturally robust to LiDAR-camera misalignments. This elegantly compensates for the point-wise distillation's sensitivity to misalignment, resolving both "missing global context" and "misalignment sensitivity" with a single component.
“Formulating future occupancy prediction as conditional denoising” converts a perceptual auxiliary task into a generative spatio-temporal self-supervised signal without any labels. This pipeline can be readily generalized to other 3D pre-training approaches requiring temporal consistency.

Limitations & Future Work¶

Point-to-pixel distillation is sensitive to camera-LiDAR calibration misalignments (as admitted by the authors). Although global context distillation and temporal diffusion partially mitigate this, the vulnerability of the local alignment term remains unresolved.
The diffusion target is built on ground-filtered BEV projections, which discards vertical information. This leads to limited gains in distinguishing ground-level semantics (e.g., terrain vs. sidewalk). The authors propose that future work could extend the diffusion beyond BEV (e.g., latent diffusion) to better preserve 3D geometry and temporal characteristics.
The multi-layer pairing currently relies on fixed heuristics (the final two layers), and both layer-to-layer matching strategies and loss weights are hand-tuned. The authors project utilizing "learnable matching + dynamic loss weighting" for future optimization.
For certain baseline methods (such as LiMoE / LiMA), the authors note that checkpoints or source codes were unavailable, posing reproducibility caveats during cross-method comparisons.

vs ScaLR / CleverDistiller: These methods perform direct feature alignment but only focus on the final layer of the VFM while throwing away the CLS token. HilDA distills multiple layers + the CLS token, capturing the layer-by-layer evolution of semantics and global context, which brings particularly large gains in data-scarce settings.
vs LiMA: LiMA previously showed distinct advantages in linear probing (owing to cross-view/long-term temporal distillation), but HilDA maintains a leading position in LP while yielding superior fine-tuning results. This indicates that representations produced by hierarchical distillation + temporal diffusion are more "multi-task transferable."
vs Discriminative Occupancy Pre-training (ALSO / GASP, etc.): These methods employ voxel-wise binary cross-entropy, only modeling local marginal distributions and lacking incentives for global consistency. HilDA uses conditional diffusion to jointly model occupancy, imposing scene-level spatio-temporal structures via iterative denoising; in the ablation study, diffusion > simple decoder > ALSO.
vs SuperFlow / LiMoE (utilizing semantic priors/grouping): These methods rely on heavy priors such as SAM/OpenSeed pseudo-masks to stabilize distillation. HilDA utilizes only geometrically calibrated point-to-pixel correspondences, achieving superior results without requiring extra grouping priors or scaled-up dataset sizes.

Rating¶

Novelty: ⭐⭐⭐⭐☆ "Hierarchical distillation + generative temporal occupancy" represents two robust and insightful upgrades to mature distillation paradigms. While individual components are not entirely net-new, their combination and motivation are clear.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Highly thorough validation covering 6 types of tasks (segmentation, detection, occupancy, scene flow, robustness, and domain generalization), three scales of teacher models, and extensive ablation studies.
Writing Quality: ⭐⭐⭐⭐☆ The motivational chain is clear, and the ablations are well-supported. Figures 1 and 2 are intuitive. Although the formula density is slightly high, each math expression holds a core mechanism.
Value: ⭐⭐⭐⭐☆ An annotation-free general LiDAR pre-training framework with clear transfer value for autonomous driving perception. The hierarchical distillation concepts are also generalizable to other cross-modal distillation scenarios.