HD2-SSC: High-Dimension High-Density Semantic Scene Completion for Autonomous Driving

Conference: AAAI 2026
arXiv: 2511.07925
Code: https://github.com/PKU-ICST-MIPL/HD2-AAAI2026
Area: Autonomous Driving
Keywords: Semantic Scene Completion, dimension gap, density gap, semantic decoupling, voxel alignment

TL;DR

This paper proposes the HD2-SSC framework. It addresses the 2D-to-3D input-output dimension gap through a High-Dimension Semantic Decoupling (HSD) module (expanding pixel features along pseudo-dimensions and orthogonally decoupling them), and addresses the annotation-reality density gap through a High-Density Occupancy Refinement (HOR) module (aligning geometric and semantic critical voxels via a "detection-refinement" paradigm), achieving SOTA results on SemanticKITTI and SSCBench-KITTI-360.

Background & Motivation

Camera-based 3D Semantic Scene Completion (SSC) is a crucial task for autonomous driving, requiring the inference of occupancy and semantic information in 3D space from 2D images. MonoScene pioneered the lifting of 2D image features to 3D volumes, and subsequent works have developed architectures like BEV, TPV, and Transformers to improve 3D scene representations.

Limitations of Prior Work: Existing methods focus on 3D feature refinement but do not differentiate between pixel features and voxel semantics during view transformation and occupancy prediction, facing two key challenges:

Dimension Gap: The input image is from a 2D planar perspective, where pixel features mix the semantics of multiple objects due to occlusion (coarse pixel semantics). SSC requires fine-grained voxel semantics from a 3D perspective, necessitating the expansion and decoupling of coarse pixel features.

Density Gap: Manual annotations from LiDAR sensors are inherently sparse (with empty spaces), whereas real-world scenes feature dense occupancy and rich contextual details, requiring the detection of missing voxels and correction of errant ones.

Key Challenge: Directly using 2D coarse pixel features for 3D prediction leads to semantic confusion and occlusion issues; sparse-annotation-guided prediction lacks density and cannot recover the actual dense occupancy.

Key Insight: Approaching the problem from two dimensions of information transformation: first, semantic expansion and decoupling during dimension conversion; second, geometric-semantic consistency alignment during density completion.

Method

Overall Architecture

HD2-SSC = Image Encoder (ResNet50+FPN to extract 2D features) → HSD Module (decouple coarse pixel semantics) → View Transformation (2D-to-3D projection) → HOR Module (refine voxel occupancy) → SSC Prediction.

Key Designs

1. High-Dimension Semantic Decoupling (HSD) Module:

   • Pseudo Voxelization:
     • Function: Expands 2D image features along a pseudo "semantic dimension" into pseudo-voxelized features.
     • Mechanism: Uses a dimension expansion (DE) layer (2D convolution) to lift \(F_{cam}\) into \(D_{exp}\) slices of pseudo-voxelized features \(F_{pseudo}\), where each slice corresponds to a potential occluded semantic class.
     • Orthogonal Loss: \(L_{orth} = \lambda \|W_{DE} \cdot W_{DE}^T - I\|\), prompting the expanded slices to have distinct semantic directions.
     • Design Motivation: A single pixel location may correspond to multiple occluded objects, necessitating expansion along a new dimension to provide multiple candidate semantics.
   • Semantic Aggregation:
     • Function: Aggregates high-dimension semantics from the pseudo-voxelized features.
     • Mechanism: (1) Pixel queries \(Q_{pixel}\) gather global semantics via cross-attention → (2) DPC-kNN semantic clustering groups global semantics into \(D_{exp}\) clusters → (3) The similarity between each pseudo-voxel slice and the clusters is calculated for weighted aggregation.
     • Decoupling Loss: \(L_{decouple} = \sum_{i \neq j} \frac{C_i \cdot C_j}{\|C_i\| \cdot \|C_j\|}\), driving cluster semantics to be as orthogonal as possible.
     • Design Motivation: Ensures different expanded dimensions capture distinct object semantics, avoiding redundancy.

2. High-Density Occupancy Refinement (HOR) Module:

   • Detection Phase:
     • Function: Comprehensively detects occupied voxels and identifies geometrically critical voxels.
     • Mechanism: A binary classification head generates two score maps—occupancy/free separation \(M_{o-f}\) + foreground/background separation \(M_{f-b}\). Summing these two maps yields a geometric density score to select the top-\(k\) geometrically critical voxels \(V_{geo}\).
     • Design Motivation: Provides coarse-grained but comprehensive occupancy detection, serving as a geometric structural prior for subsequent refinement.
   • Refinement Phase:
     • Function: Predicts multi-class semantics and identifies semantically critical voxels.
     • Mechanism: A multi-class classification head generates the initial SSC prediction \(Y_{init}\), selecting the top-\(k\) semantically critical voxels \(V_{sem}\) based on classification confidence.
     • Design Motivation: Identifies the most discriminative voxels from a semantic perspective.
   • Voxel Alignment:
     • Function: Aligns the distributions of geometric and semantic critical voxels.
     • Mechanism: Uses symmetric KL divergence to align the distributions of \(V_{geo}\) and \(V_{sem}\), then adds the aligned critical voxel information residuals to the initial prediction via an MLP.
     • Refinement Equation: \(Y_{refine} = Y_{init} + \text{MLP}([V_{geo}, V_{sem}])\)
     • Design Motivation: Ensures consistency between geometric and semantic structures, completing missing voxels while correcting errant ones.

Loss & Training

• Three auxiliary losses: orthogonal loss \(L_{orth}\), decoupling loss \(L_{decouple}\), and critical voxel alignment loss \(L_{critical}\).
• Training: 24 epochs, 4×A6000 GPUs, batch size of 4.
• AdamW optimizer, learning rate of \(2 \times 10^{-4}\), weight decay of \(1 \times 10^{-2}\).
• Expansion dimension \(D_{exp} = 4\), number of queries \(N_{query} = 100\), number of critical voxels \(k = 4096\).
• Feature resolution: 2D is 1/16 of the input, 3D is \(128 \times 128 \times 16\) upsampled to \(256 \times 256 \times 32\).

Key Experimental Results

Main Results (SemanticKITTI Validation Set)

Method	SC IoU↑	SSC mIoU↑
VoxFormer	44.15	13.35
HASSC	44.58	14.74
Symphonies	41.92	14.89
CGFormer	45.99	16.87
SGN	46.21	15.32
HD2-SSC (Ours)	47.59	17.44

SSCBench-KITTI-360 Test Set

Method	SC IoU↑	SSC mIoU↑
CGFormer	48.07	20.05
SGN	47.06	18.25
Symphonies	44.12	18.58
HD2-SSC (Ours)	48.58	20.62

Ablation Study

Configuration	IoU↑	mIoU↑	Description
Baseline (VoxFormer)	44.15	13.35	-
+ HSD	46.45	15.58	IoU+2.30, mIoU+2.23
+ HOR	46.07	16.12	IoU+1.92, mIoU+2.77
+ HSD + HOR	47.59	17.44	Best complementary performance

Loss Function Ablation

Configuration	IoU↑	mIoU↑
HD2-SSC (full)	47.59	17.44
w/o \(L_{orth}\)	46.93 (-0.66)	16.64 (-0.80)
w/o \(L_{decouple}\)	46.85 (-0.74)	16.78 (-0.66)
w/o \(L_{critical}\)	46.49 (-1.10)	16.31 (-1.13)

Key Findings

• HOR contributes more to mIoU (+2.77 vs +2.23 for HSD): implying that the density gap is a more critical limiting factor for semantic scene completion performance.
• HSD contributes more to IoU (+2.30 vs +1.92 for HOR): suggesting that dimension decoupling provides a more direct benefit to overall geometric completion.
• \(L_{critical}\) is the most critical loss: its removal leads to a drop of 1.10 in IoU and 1.13 in mIoU, which is significantly larger than the drops caused by removing the other two losses.
• Expansion dimension \(D_{exp}=4\) is optimal: further increases introduce "virtual" semantics that do not correspond to actual objects, thereby reducing performance.
• Superior efficiency compared to SGN: with only 0.8M more parameters, it uses less GPU memory (14.42G vs 15.83G) and achieves faster inference (0.56s vs 0.61s), thanks to operating on a \(128^3\) feature grid which avoids SGN's upsampling overhead.
• Occ3D-nuScenes generalization: IoU 75.4, mIoU 44.2, which outperforms OccFormer (70.1/37.4) and BEVDet4D (73.8/39.3).

Highlights & Insights

• Precise problem definition: Clearly identifies the two overlooked fundamental issues of "dimension gap" and "density gap," rather than simply stacking architectural modules.
• Clever use of orthogonal loss: Elegantly ensures the diversity of different semantic slices by constraining the orthogonality of the expansion layer's weight matrix.
• Two-stage detection-refinement design: Coarse-to-fine scheme; geometrically critical voxels provide structural priors, while semantically critical voxels provide class priors. KL divergence alignment ensures consistency.
• Balancing efficiency and performance: Operating on a smaller feature grid (\(128^3\)) yields superior performance and faster inference.

Limitations & Future Work

• Failure cases still occur in severely occluded and far-range regions (e.g., incorrect occupancy predictions and incomplete boundaries).
• Pseudo-voxelization lacks explicit pixel-wise semantic label supervision; hence, the semantics of the expanded dimensions may lack precision.
• Validation is primarily restricted to the KITTI series, with only preliminary validation on nuScenes.
• The expansion dimension \(D_{exp}=4\) is manually selected; adaptive dimension selection is worth exploring.
• Lacks in-depth comparison with the latest 3D Gaussian-based methods (such as GaussianFormer).
• Incorporating physical regularization in the future could potentially complement the semantic features of low-quality regions.

Related Work & Insights

• Evolution from MonoScene to VoxFormer: Moving from dense volumetric projection to a two-stage approach (visible region aggregation + full scene propagation). Based on this, HD2-SSC targets the overlooked dimension and density gaps.
• SGN's dense-sparse-dense strategy: Complementary to HD2-SSC's concept—while SGN dynamically selects discriminative voxels, HD2-SSC decouples pixel semantics and aligns critical voxels.
• Broad application of orthogonal loss in representation learning: The concept of using orthogonal constraints to encourage diverse representations can be generalized to other scenarios requiring feature decoupling.
• Insight: In any task involving dimensional transformation (e.g., 2D-to-3D, text-to-image), explicitly considering the input-output information gap (dimension, density, resolution, etc.) may yield greater efficiency than merely refining intermediate representations.

Rating

• Novelty: ⭐⭐⭐⭐ (Innovative formulation of the dimension gap and density gap; robust design of HSD+HOR)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive comparisons on two benchmarks, thorough ablation studies, efficiency analysis, generalization validation, and failure case analysis)
• Writing Quality: ⭐⭐⭐⭐ (Clear motivation and problem statement, with rich architectural diagrams and visualizations)
• Value: ⭐⭐⭐⭐ (An effective methodological contribution to the autonomous driving SSC domain, achieving SOTA on two datasets)

Related Papers

• [AAAI 2026] Towards 3D Object-Centric Feature Learning for Semantic Scene Completion
• [AAAI 2026] ReflexDiffusion: Reflexion-Enhanced Trajectory Planning for High Lateral Acceleration in Autonomous Driving
• [AAAI 2026] Unleashing Semantic and Geometric Priors for 3D Scene Completion
• [CVPR 2026] OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective
• [NeurIPS 2025] X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability