Skip to content

title: >- [Paper Note] DeGuNet: Depth-Guided Ultra-Compact Backbones for Efficient LiDAR-Camera 3D Detection description: >- [ECCV 2026][Autonomous Driving][3D Object Detection] Addressing the severe parameter redundancy and geometric misalignment of 2D-pretrained visual backbones in multi-modal 3D detection, DeGuNet proposes an ultra-compact (0.31M parameters) depth-guided image backbone equipped with MPIR blocks and MMViT attention, reducing GPU memory by 66.5% while delivering up to 6.20 absolute mAP gain. tags: - ECCV 2026 - Autonomous Driving - 3D Object Detection - LiDAR-Camera Fusion - Lightweight Backbone date: 2026-09-19 content_hash: 3536997822ca3e79

DeGuNet: Depth-Guided Ultra-Compact Backbones for Efficient LiDAR-Camera 3D Detection

Conference: ECCV 2026
Paper: ECCV 2026 Paper Page
Code: Pending
Area: Autonomous Driving
Keywords: 3D Object Detection, LiDAR-Camera Fusion, Sparse Representation, Lightweight Backbone, Depth Guidance

TL;DR

DeGuNet replaces parameter-heavy, 2D-pretrained visual backbones with an ultra-compact 0.31M depth-guided backbone that leverages masked partial convolutions (MPIR) and masked attention (MMViT) to prevent sparse LiDAR projection contamination, reducing GPU memory by 66.5% and accelerating inference while gaining up to 6.20 mAP across established multi-modal 3D detectors.

Background & Motivation

In autonomous driving perception, multi-modal 3D object detection fusing LiDAR point clouds and multi-view camera images has established itself as the dominant architectural paradigm. While LiDAR provides centimeter-accurate 3D geometric spatial structure, surrounding multi-camera rigs supply critical high-resolution color textures, fine-grained contours, and semantic context. Prevailing state-of-the-art frameworks (such as BEVFusion, DeepInteraction, and EA-LSS) typically deploy two separate modality streams before projecting and aggregating cross-modal features onto a unified Bird's-Eye-View (BEV) representation space. However, this foundational pipeline remains overwhelmingly tethered to bulky, over-parameterized visual backbones (e.g., deep ResNets or Swin Transformers) pretrained on conventional 2D semantic tasks like ImageNet classification or 2D instance segmentation. Empirical profiling reveals that standard image backbones alone swallow between 30M and 78M parameters, consuming anywhere from 50% to 86.6% of the entire system's parameter budget. This constitutes severe system-level resource redundancy for a camera modality that primarily plays a complementary role to the LiDAR stream.

Beyond sheer memory footprint and latency overhead, a fundamental structural tension plagues this paradigm: standard 2D semantic priors are inherently misaligned with the spatial demands of 3D BEV perception. Features optimized for 2D classification lack metric depth awareness and cross-view spatial consistency, both of which are foundational prerequisites for accurate 3D bounding box regression. While an intuitive recourse is shifting the visual backbone pretraining objective from 2D classification to LiDAR-guided depth completion, empirical results demonstrate that simply fine-tuning conventional lightweight backbones (such as MobileViT or tiny ResNets) on depth completion yields plateaued downstream detection accuracy (around 62.1 mAP). The root cause lies in the extreme spatial sparsity of projected LiDAR depth. When 3D point clouds are mapped onto perspective camera planes, over 98% of the pixels remain empty, unmeasured voids. Standard spatial convolutional operators apply translation-invariant kernels across the entire grid, indiscriminately blending sparse valid geometric signals with vast swathes of zero-value regions. As the network deepens and receptive fields expand, fragile geometric cues are diluted and corrupted by invalid background tokens.

Therefore, bridging this semantic-geometric divide demands far more than an objective-level swap; it necessitates purpose-built, sparsity-aware neural operators. The core idea is to construct DeGuNet, an ultra-compact (0.31M parameter) plug-and-play image backbone designed for depth-guided representation learning, which employs dual-branch Masked Partial Inverted Residuals (MPIR) and Masked Mobile Vision Transformer (MMViT) blocks to strictly quarantine invalid projection regions while progressively injecting dense RGB semantics into valid geometric anchors prior to cross-modal BEV fusion.

Method

Overall Architecture

DeGuNet operates as a general-purpose, drop-in feature extraction infrastructure designed to seamlessly replace heavyweight 2D backbones across diverse LSS-based (Lift-Splat-Shoot) multi-modal 3D detection frameworks. The input consists of synchronized multi-view RGB camera images alongside corresponding sparse LiDAR depth projection maps. To decouple geometric alignment from downstream detection overhead, DeGuNet follows a distinct two-phase training lifecycle: 1. Phase 1: Geometry-Guided Pretraining: A lightweight encoder-decoder architecture is formulated for dense depth completion. The encoder (the core DeGuNet network) processes dense multi-view RGB images and sparse LiDAR projections through three hierarchical downsampling stages (\(1/2\), \(1/4\), and \(1/8\) resolutions) to extract geometry-aligned representations; an auxiliary spatial upsampling decoder reconstructs multi-scale dense depth predictions solely to compute masked depth loss against ground-truth point clouds. 2. Phase 2: End-to-End Detection Integration: Upon pretraining completion, the auxiliary depth decoder is discarded entirely. The pretrained DeGuNet encoder is plugged directly into the downstream multi-modal 3D detector as the primary image backbone. Densified, depth-aware multi-scale features are channeled through a lightweight LiteNeck that decouples depth logits from visual context features, which are then projected via standard LSS view transformers into the shared BEV space to be fused with voxelized LiDAR features.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View RGB Images + Sparse LiDAR Projections"] --> B["Stage 1: MPIR Local Feature Extraction<br/>Dual-branch masked partial convolutions isolate zero-value voids"]
    B --> C["Stage 2: Progressive Cross-Modal Guidance<br/>Dense RGB semantics injected strictly into valid geometric anchors"]
    C --> D["Stage 3: MMViT Global Context Modeling<br/>Mask-aware self-attention blocks invalid background tokens"]
    D --> E["LiteNeck Feature Decoupling<br/>Separates categorical depth logits from visual context features"]
    E --> F["LSS Frustum Back-Projection & BEV Multi-Modal Fusion Detection"]

Key Designs

1. MPIR Local Feature Extraction: Isolating Invalid Voids via Partial Convolutions Standard convolutional layers aggregate values across their entire receptive field indiscriminately, causing the >98% zero-value empty pixels of projected LiDAR to dilute valid depth activations. The Masked Partial Inverted Residual (MPIR) block addresses this through a decoupled dual-branch architecture. The dense RGB stream (\(f_{\text{img}}\)) utilizes standard depth-wise separable convolutions to capture continuous semantic textures and high-frequency edges. Concurrently, the sparse LiDAR branch (\(f_{\text{lidar}}\)) implements mask-guided partial convolutions. Guided by a binary validity mask \(f_{\text{mask}}\) indicating the exact spatial coordinates of valid LiDAR points, partial convolutions confine kernel weight updates and feature aggregation strictly to valid pixels, dynamically renormalizing outputs based on valid pixel density. Subsequent pooling operations update the binary mask \(f_{\text{mask}}\) for succeeding layers, guaranteeing that expanding receptive fields do not blur geometric boundaries with zero-value noise.

2. Progressive Cross-Modal Guidance: Mask-Bounded Semantic Injection Extracting features solely from sparse LiDAR points severely restricts feature density and downstream expressive capacity. To enrich geometric representations without leaking semantic hallucination into unmeasured regions, DeGuNet introduces progressive Guide modules at the early \(1/2\) and \(1/4\) resolution stages. The Guide module concatenates extracted image features \(f_{\text{img}}\) and sparse geometric features \(f_{\text{lidar}}\), applies a \(3 \times 3\) convolution followed by Batch Normalization (BN) and ReLU activation, and strictly enforces an element-wise multiplication with the downsampled geometric validity mask: $\(f_{\text{out}} = \sigma\left(\text{BN}\left(\text{Conv}\left([f_{\text{img}}, f_{\text{lidar}}]\right)\right)\right) \odot f_{\text{mask}}\)$ where the binary mask is recursively updated across downsampling stages via max-pooling: \(f_{\text{mask}}^{(l+1)} = \text{MaxPool}_{s \times s}(f_{\text{mask}}^{(l)})\) with downsampling stride \(s\). This structural gating mathematically restricts dense color semantics from bleeding into invalid spatial regions, ensuring that early cross-modal fusion densifies only structurally verified physical surfaces.

3. MMViT Global Context Modeling: Mask-Aware Self-Attention At deeper, low-resolution stages (\(1/8\) resolution), global topological context and long-range structural dependencies must be captured across objects. However, standard Transformer self-attention computes dense pairwise affinities across all spatial tokens, allowing unmeasured background positions to pollute foreground geometric representations. DeGuNet incorporates the Masked Mobile Vision Transformer (MMViT) block. The downsampled binary mask \(f_{\text{mask}}\) is dynamically flattened and injected into the attention affinity computation. By adding a large negative penalty (\(-\infty\)) to invalid token coordinates prior to Softmax normalization: $\(\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d}} + \mathcal{M}_{\text{mask}}\right) V\)$ where \(\mathcal{M}_{\text{mask}} \to -\infty\) for invalid pixel locations. This formulation guarantees that attention weights corresponding to empty background nodes converge to zero, isolating global token aggregation exclusively to valid geometric anchors and preserving spatial fidelity.

4. LiteNeck Feature Decoupling: Plug-and-Play LSS Adaptation To enable seamless drop-in integration into any existing LSS-based detection framework without altering downstream view transform or detection head logic, DeGuNet employs a streamlined LiteNeck module at the encoder output. Bypassing heavy multi-scale FPN structures, LiteNeck cleanly splits the densified multi-scale geometric feature representation into two decoupled tensors along the channel dimension: categorical Depth Logits for frustum depth probability distribution modeling, and Context Features for downstream instance recognition. Downstream detectors can directly feed these outputs into standard Lift-Splat-Shoot pipelines, maintaining total modularity.

Loss & Training

During Phase 1 pretraining, the spatial upsampling decoder generates multi-scale depth maps \(\{\hat{Z}^{(i)}\}_{i=1}^K\). Supervised exclusively against the ground-truth sparse LiDAR depth \(Z\) filtered by validity mask \(M = \mathbb{I}[Z > 0]\), the network minimizes a multi-scale \(L_1\) depth completion loss: $\(\mathcal{L}_{dc} = \sum_{i=1}^K \gamma^{K-i} \left\| (\hat{Z}^{(i)} - Z) \odot M \right\|_1\)$ where \(\gamma\) denotes the scale-weighting factor across hierarchical stages. Gradients back-propagate exclusively through valid points, preventing false penalties on unmeasured sky or distant backgrounds. In Phase 2 downstream detector training, the auxiliary depth decoder and \(\mathcal{L}_{dc}\) are discarded; DeGuNet is fine-tuned end-to-end alongside the multi-modal detector using the baseline's official detection losses, optimizers, and learning rate schedules.

Key Experimental Results

Main Results

On the nuScenes validation benchmark, DeGuNet was integrated into five representative multi-modal 3D detection frameworks. System efficiency was evaluated via Parameter Efficiency (Eff. = \(\frac{\text{mAP}_{\text{multi}} - \text{mAP}_{\text{lidar}}}{\text{Image Params (M)}}\)), quantifying detection gain per million image parameters.

Method Image Backbone Image Params ↓ Total Params ↓ Image Param Ratio ↓ mAP ↑ NDS ↑ Param Efficiency (Eff.) ↑
BEVFusion (2022) [26] CBSwin-T 78.1M 90.2M 86.61% 69.60 72.1 0.06
BEVFusion (2023) [30] Swin-T 31.8M 40.8M 77.94% 68.50 71.4 0.12
EA-LSS (2023) [18] CBSwin-T 78.33M 153.7M 50.96% 70.90 72.8 0.08
GraphBEV (2024) [42] Swin-T 31.8M 42.8M 74.30% 70.10 72.9 0.17
IS-Fusion (2024) [56] Swin-T 29.1M 48.8M 59.63% 71.00 72.7 0.20
BEVFusion (2022) + DeGuNet DeGuNet 1.11M 15.3M (-74.9M) 7.25% 70.30 (+0.70) 72.5 (+0.4) 4.86
BEVFusion (2023) + DeGuNet DeGuNet 0.31M 11.1M (-29.7M) 2.79% 69.40 (+0.90) 72.7 (+1.3) 15.29
EA-LSS + DeGuNet DeGuNet 17.0M 92.1M (-61.6M) 18.46% 71.10 (+0.20) 72.9 (+0.1) 0.36
GraphBEV + DeGuNet DeGuNet 0.81M 12.2M (-30.6M) 6.64% 70.40 (+0.30) 72.9 (+0.0) 7.09
IS-Fusion + DeGuNet DeGuNet 0.31M 19.7M (-29.1M) 1.57% 71.30 (+0.30) 73.0 (+0.3) 20.00

Inference runtime and peak memory metrics measured on an NVIDIA A100 GPU (batch size 1) demonstrate radical deployment advantages:

Method Inference FPS ↑ GPU Memory ↓ Image Backbone Latency ↓ Total Image Pipeline Latency ↓ Speedup Ratio ↑ Memory Reduction Ratio ↓
DeepInteraction [54] 0.2 43.11 GB 1201.6 ms 2162.7 ms 0.045× -110.8%
BEVFusion (2022) [26] 0.3 41.01 GB 1258.1 ms 2236.3 ms 0.068× -100.5%
BEVFusion (2023) [30] 4.4 20.46 GB 21.12 ms 51.58 ms 1.0× (Baseline) -
DeGuNet (Ours) 5.1 6.86 GB 7.59 ms 40.76 ms 1.16× 66.5%

Ablation Study

A component-wise ablation study conducted on the BEVFusion (2023) baseline traces the cumulative performance contributions starting from the LiDAR-only model (64.66 mAP / 10.30M parameters):

Configuration Detection mAP ↑ Gain (∆mAP) ↑ System Parameters ↓ Note
LiDAR Baseline 64.66 - 10.30M Single-modality LiDAR baseline
+ MPIR blocks 66.14 +1.48 10.32M Masked partial convolutions prevent void pollution
+ Guide modules 67.24 +2.58 10.41M Progressive multi-scale cross-modal RGB injection
+ MMViT blocks 68.38 +3.72 10.61M Mask-aware attention eliminates background tokens
+ LiteNeck (Full DeGuNet) 69.40 +4.74 11.10M Full model with depth/context feature decoupling
Full w/o MPIR 66.59 -2.81 11.13M Removing partial convolutions causes sharp degradation
Full w/o Guide 66.14 -3.26 11.05M Removing semantic guidance drops performance most (-3.26)
Full w/o MMViT 67.24 -2.16 10.95M Omitting masked attention limits long-range dependencies
Full (All Standard Conv) 68.01 -1.39 11.10M Reverting to standard conv degrades geometric fidelity

Ablating cross-modal fusion stages confirms that single-stage semantic injection achieves modest improvements (\(1/2\) stage only: +0.44 mAP; \(1/4\) stage only: +0.75 mAP), whereas progressive multi-scale injection across all three stages maximizes structural densification, achieving optimal performance at 67.24 mAP (+1.10 mAP over the unguided baseline).

Key Findings

  • Cross-Modal Guidance is the Primary Driver: Removing the progressive Guide module incurs the steepest accuracy penalty (-3.26 mAP, dropping from 69.40 to 66.14), proving that sparse geometric structures require dense, high-resolution visual semantics to unlock discriminative representation density.
  • Sparsity-Aware Operators Outperform Heavy Standard Models: In KITTI depth pretraining, even the 31.82M Swin-T yields poor depth RMSE (1.83 m) and downstream detection mAP (64.01), whereas DeGuNet's 0.31M parameters achieve 0.74 m RMSE and 69.40 mAP. Replacing DeGuNet's masked operators with standard convolutions causes an immediate 1.39 mAP drop, proving that architectural sparsity awareness is strictly indispensable.
  • Transformative Parameter Efficiency: On IS-Fusion, DeGuNet slashes image-side parameters from 29.1M to 0.31M, boosting the parameter efficiency index from 0.20 to 20.00 (a 100× improvement) while pruning peak GPU memory by 66.5% (down from 20.46 GB to 6.86 GB), shattering the prevailing assumption that multi-modal fusion requires heavy 2D vision backbones.

Highlights & Insights

  • Exposing the Architectural Redundancy of 2D Priors: The paper convincingly demonstrates that over 80% of parameters in modern multi-modal 3D detectors reside in 2D image backbones that yield marginal parameter efficiency (<0.30), exposing an inherent disconnect between 2D semantic objectives and 3D BEV geometric requirements.
  • Zero-Cost Sparsity Quarantine: The coupling of MPIR's partial convolutions and MMViT's negative-infinity attention masking provides an exceptionally clean mathematical guarantee against zero-value dilution without introducing heavy computational parameters.
  • Universal Plug-and-Play Adaptability: By decoupling depth distribution logits from context features via LiteNeck, DeGuNet integrates seamlessly into existing LSS-based architectures (BEVFusion, EA-LSS, GraphBEV) as an out-of-the-box replacement with zero downstream code modification.

Limitations & Future Work

  • Author-Acknowledged Bottlenecks: Due to its ultra-compact 0.31M parameter budget, DeGuNet's capacity for high-level semantic abstraction remains inherently constrained when distinguishing rare, long-tail categories or atypical obstacle classes.
  • Sensor Calibration Sensitivity: The architecture fundamentally assumes accurate extrinsic projection to construct the binary mask \(f_{\text{mask}}\). Under calibration drift or severe weather conditions (e.g., dense rain/fog causing point dropouts or spurious returns), inaccurate validity masks could inadvertently suppress valid visual signals.
  • Promising Research Directions: Future investigations could explore distilling semantic knowledge from massive vision foundation models into DeGuNet to bolster long-tail recognition, alongside designing soft, probabilistic mask filtering mechanisms to handle noisy sensor calibrations.
  • vs. BEVFusion / Swin-T: BEVFusion employs a heavy 2D Swin-T backbone (31.8M-78.1M parameters) pretrained on ImageNet classification. DeGuNet swaps it for a 0.31M geometry-pretrained backbone, trimming GPU memory by 66.5% while improving detection accuracy by 0.70-0.90 mAP.
  • vs. BEVDepth (Auxiliary Depth Supervision): BEVDepth appends auxiliary depth supervision heads to the downstream detector, increasing detection training memory and leaving the bulky 2D backbone untouched. DeGuNet completely decouples depth completion into pretraining, eliminating all auxiliary depth prediction overhead during downstream inference.
  • vs. Submanifold Sparse Convolutions (VoxelNeXt / SpConv): While submanifold sparse convolutions optimize irregular 3D point cloud processing in voxel grids, DeGuNet extends sparsity-aware principles to perspective multi-view camera planes, effectively conquering projection sparsity and zero-value dilution.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the first ultra-compact, depth-guided backbone explicitly tailored to overcome spatial sparsity in multi-modal 3D detection.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 established fusion baselines with rigorous efficiency, latency, and ablation breakdowns.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression, self-contained mathematical formulations, and polished visual flowcharts.
  • Value: ⭐⭐⭐⭐⭐ Delivering 66.5% GPU memory reduction and superior mAP with only 0.31M parameters offers immense practical value for resource-constrained autonomous driving hardware.