StreamOcc: Streaming Dense Voxel Representations for 3D Occupancy Prediction¶

Conference: ECCV 2026
arXiv: 2503.22087
Code: TBD
Area: Autonomous Driving
Keywords: 3D Occupancy Prediction, Streaming Processing, Dense Voxel, Dynamic Object Modeling, Temporal Fusion

TL;DR¶

StreamOcc integrates the streaming update paradigm with dense 3D voxel representations for occupancy prediction for the first time. It addresses the challenges of warping distortion and dynamic object feature degradation through rectified streaming voxel aggregation (StreamAgg) and query-guided aggregation (QueryAgg), achieving state-of-the-art (SOTA) performance on SurroundOcc and Occ3D-nuScenes with a real-time latency of 83.3ms/frame and a memory footprint of 2.8GB.

Background & Motivation¶

Vision-based 3D occupancy prediction is a core task in autonomous driving perception. Its goal is to classify each voxel in space into static objects (e.g., road surfaces, sidewalks), dynamic objects (e.g., vehicles, pedestrians), or free space, providing a dense 3D scene understanding for downstream planning. Existing mainstream methods can be divided into two categories: one uses dense voxel representations to preserve fine-grained 3D spatial details, but requires processing multi-frame historical voxel features simultaneously during inference. This results in a massive memory overhead of 5-12GB and a latency of 166-1250ms, which is far from meeting the deployment requirements of in-vehicle systems. The other category shifts toward sparse representations (such as sparse voxel streams, Gaussian primitives, or compressed BEV/tri-plane features) to improve efficiency. However, compression inherently sacrifices spatial fidelity, creating a ceiling for fine-grained semantic modeling—especially for distant pedestrians and occluded vehicles, where sparse representations often yield only blurred outlines.

The streaming paradigm has recently demonstrated strong temporal processing efficiency in sparse prediction tasks such as 3D object detection and map detection. Instead of repeatedly processing multi-frame inputs, it performs lightweight fusion between the recursively propagated feature from the previous frame and the current frame feature, which greatly saves computation while maintaining temporal consistency. However, extending this paradigm to dense voxel-based 3D occupancy prediction suffers from two fundamental challenges. First, aligning voxel features across frames requires warping via trilinear interpolation to compensate for ego-motion. This interpolation inevitably introduces distortions (such as numerical dispersion and boundary blurring) at object boundaries, which are continuously amplified during recursive accumulation. Second, during the projection from images to voxels, dynamic objects (vehicles, pedestrians) suffer from severe information loss due to motion misalignment, sparse pixel projection at long distances, feature mixing during multi-object overlap, and projection truncation caused by occlusion. Yet, these dynamic objects are precisely the most critical for autonomous driving safety. Existing multi-frame dense methods are difficult to deploy in real-time due to high overhead, while streaming sparse methods lose dense spatial accuracy. So far, no approach has established a path to simultaneously obtain the fine details of dense voxels and the high efficiency of the streaming paradigm.

Key Insight: Rather than discarding the precision advantages of dense voxels, it is better to directly confront the two aforementioned challenges and design targeted aggregation strategies for each.

Core Idea: This paper proposes the StreamOcc framework, which deeply integrates streaming updates with dense voxel representations. Specifically, it employs the StreamAgg module to eliminate interpolation-induced distortion and semantic drift via motion-aware warping coupled with adaptive residual rectification. Additionally, it utilizes the QueryAgg module to extract dynamic object semantics from the image space using instance queries and selectively injects them into the corresponding occupied voxel regions. This achieves a substantial improvement in both overall and dynamic object occupancy prediction accuracy while maintaining real-time performance (83.3ms/frame, 2.8GB GPU memory).

Method¶

Overall Architecture¶

StreamOcc is a streaming occupancy prediction framework utilizing a two-stage aggregation scheme. First stage (StreamAgg) is responsible for recursive temporal fusion and distortion rectification of voxel features: the multi-view images of the current frame are processed by ResNet-FPN to extract 2D features, which are then projected into the 3D voxel space to obtain the current voxel features. Meanwhile, the voxel features propagated from the previous frame are first aligned to the current ego-coordinate system via motion-aware warping, refined by an adaptive residual rectification module to eliminate warping-induced distortions, and finally concatenated with current voxel features and fused via 1D convolution to produce temporally consistent aggregated voxel features. Stage two (QueryAgg) specifically targets dynamic object enhancement: a Sparse4Dv3 detector generates instance queries from the image space, injects spatial-geometric information carried by the voxels into the queries via voxel-to-query aggregation to resolve depth ambiguity, and then selectively writes back the rich semantic features from the queries into the voxel regions occupied by dynamic objects through dynamic query aggregation. Finally, a dense 3D semantic occupancy prediction is output via an MLP decoder.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Image Sequence"] --> B["2D→3D Voxel Projection<br/>ResNet+FPN"]
    B --> C["Current Voxel Features"]
    D["Previous Voxel Feature<br/>V_{t-1}"] --> E["Motion-Aware Warp<br/>Coordinate Alignment"]
    E --> F["Adaptive Residual Rectification<br/>Remove Warping Distortion"]
    F --> G["Rectified Warp Feature"]
    C --> G
    G --> H["Conv1D Fusion<br/>StreamAgg"]
    H --> I["Query-Guided Aggregation<br/>QueryAgg"]
    J["Sparse4Dv3<br/>Instance Query Detector"] --> I
    I --> K["Final Voxel Feature"]
    K --> L["MLP Occupancy Head"]
    L --> M["3D Occupancy Prediction"]

Key Designs¶

1. StreamAgg: Streaming Voxel Temporal Aggregation with Distortion Rectification

The streaming recursive fusion of dense voxel features faces a foundational challenge: ego-motion compensation during feature propagation relies on trilinear interpolation to resample the previous frame's voxel features. However, interpolation inevitably introduces boundary blurring and numerical dispersion, and these distortions are continuously amplified during recursive accumulation. StreamAgg's solution is to insert a "rectification gateway" after warping: first, the voxel coordinates of the previous frame are mapped to the current coordinate system using the ego-motion transformation matrix, and coarsely aligned warped features are obtained via trilinear interpolation. Subsequently, the warped features are sent to the adaptive residual rectification module for fine-grained restoration. Finally, the restored warped features are concatenated with the current voxel features and compressed via 1D convolution into a voxel feature with the same number of channels as the current frame's output. The key to this design lies in forming a three-step pipeline of "align first, rectify next, and fuse last"—where rectification is deliberately placed between warping and fusion to ensure that features entering the next recursive loop are normalized beforehand, thereby breaking the propagation chain of distortion accumulation.

2. Adaptive Residual Rectification: Selective Distortion Restoration Guided by Dual Geometric and Semantic Supervision

The distortion introduced by warping is not uniformly distributed throughout the voxel space but is instead concentrated near object boundaries. To address this characteristic, this module employs a 3D channel-spatial attention residual structure: the warped features are first passed through a 3D convolutional bottleneck (compressed to C/4 and then expanded back to C) to generate rectification candidates. Then, 3D channel attention weights \(M_c\) and 3D spatial attention weights \(M_s\) are calculated on these candidates. \(M_s\) acts as a soft gating signal to weight the rectification candidates, which are then added residually to the original warped features:

\[V_{refwarp} = \sigma(M_s) \odot (M_c \odot V_{out}) + V_{warp}\]

Even more ingenious is the dual supervision mechanism. During training, two auxiliary branches are introduced: Geometry Supervision uses BCE loss of the binary occupancy mask to supervise \(M_s\), forcing spatial attention to learn to distinguish "occupied regions vs. free space," thereby focusing rectification resources on the occupied regions that actually need repair. Semantic Supervision uses the cross-entropy loss of the semantic occupancy distribution of the current frame to supervise the rectified features \(V_{refwarp}\), ensuring that its secrets align with the current scene rather than retaining residual information from the previous frame. The two supervision branches teach the model "where to attend" and "what to produce" respectively, with zero additional overhead during inference.

3. QueryAgg: Instance-Query Guided Targeted Enhancement of Dynamic Objects

While temporal accumulation of voxel features is highly effective for static scenes (e.g., roads, buildings, vegetation), it is severely deficient for dynamic objects (e.g., cars, pedestrians). This deficiency stems from the projection from images to voxels, where pixel projections for distant small objects are extremely sparse, features of close multi-objects mix within coarse voxel grids, and occlusions lead to incomplete projections—losses that cannot be compensated for by the voxel space itself. QueryAgg introduces an independent "external information source": it utilizes a streaming Sparse4Dv3 detector to generate instance queries with rich semantics from the image space, then performs feature fusion in two steps. Step one is voxel-to-query aggregation (V2Q): it allows queries to sample geometric context from the voxel features output by StreamAgg via deformable attention, helping to eliminate false-positive queries generated by depth ambiguity. Step two is dynamic query aggregation (DQA): it only filters instance queries with confidence scores higher than 0.3. For each voxel, it checks whether it falls within the predicted bounding box of any query. If there is a hit, the query features are weighted and aggregated into the voxel using cross-attention, with the injection intensity controlled by a learnable gate \(g_i\). Otherwise, the original features remain unchanged, ensuring that static regions are not interfered with:

\[\mathbf{V}_{DQA}^i = \begin{cases} \mathbf{V}_{S.A}^i, & \mathcal{N}^i = \emptyset \\ \mathbf{V}_{S.A}^i + \mathbf{g}^i \odot \mathbf{z}^i, & \text{otherwise} \end{cases}\]

The query selection strategy during training also particularly addresses the unreliability of IoU for small targets. It uses IoU + confidence filtering for large targets, but switches to geometric deviation (center distance + size deviation) coupled with confidence filtering for small targets.

Loss & Training¶

The total loss function consists of six weighted terms: the depth loss \(\mathcal{L}_{depth}\) from BEVDepth, the cross-entropy loss for occupancy prediction \(\mathcal{L}_{occ}\), the detection loss from Sparse4Dv3 \(\mathcal{L}_{det}\), the mask loss from the Auxiliary Mask Decoder \(\mathcal{L}_{mask}\), the cross-entropy loss of the semantic supervision branch \(\mathcal{L}_{sem}\), and the binary cross-entropy loss of the geometry supervision branch \(\mathcal{L}_{geo}\). The weights are empirically set as: \(\lambda_{occ}=10.0\), \(\lambda_{sem}=10.0\), \(\lambda_{geo}=10.0\), \(\lambda_{det}=0.2\), \(\lambda_{mask}=1.0\), \(\lambda_{depth}=0.05\). An AdamW optimizer is employed with input images sized \(256 \times 704\), a ResNet-50 backbone, an initial learning rate of \(2\times 10^{-4}\), and a batch size of 8. The model is trained for 24 epochs on Occ3D-nuScenes and 20 epochs on SurroundOcc.

Key Experimental Results¶

Main Results¶

Comparison with real-time methods on Occ3D-nuScenes (latency within ~100ms, measured on an A100 GPU):

Method	Backbone	mIoU	mIoUD	Latency (ms)	Memory (MB)
ViewFormer	ResNet-50	39.6	33.3	102.0	3,103
FB-OCC	ResNet-50	39.1	34.3	97.1	9,632
GSD-Occ	ResNet-50	39.4	35.1	50.0	4,759
ALOcc-mini	ResNet-50	40.6	35.6	33.1	2,577
StreamOcc	ResNet-50	41.9	38.1	83.3	2,788

On the SurroundOcc benchmark (measured on an RTX 4090): mIoU 23.4 (vs. 21.9 for the runner-up GaussianWorld), mIoUD 21.0 (vs. 19.0), IoU 33.8 (vs. 33.0), latency of 84ms, and memory of 2,788MB (2.5 times more memory-efficient than GaussianWorld).

Ablation Study¶

Configuration	mIoU	mIoUD	Latency (ms)	Memory (MB)
Single-frame Baseline	36.8	32.6	-	-
+ StreamAgg	40.4	35.4	49.0	2,437
+ StreamAgg + Det Head (Indirect Supervision)	40.8	36.3	-	-
+ StreamAgg + QueryAgg (Full)	41.9	38.1	83.3	2,788

Ablation inside adaptive residual rectification: Naive streaming 38.72 → + residual rectification (unsupervised) 39.84 → + semantic supervision 40.25 → + geometry supervision 40.37. The gains from each step are complementary, with only an additional +5ms/+14MB in total latency/memory. Query selection strategy ablation: IoU only 39.9 → + confidence 41.3 → + small target geometric constraint 41.9.

Key Findings¶

StreamAgg is the largest single-point contributor (+3.6 mIoU), proving that warping distortion rectification is a prerequisite for streaming dense voxels to be effective.
Indirect supervision (adding a detection head) provides limited improvement for dynamic objects (+0.9 mIoUD), while the direct instance feature injection of QueryAgg brings a qualitative leap (+2.7 mIoUD).
Compared with global spatial cross-attention approaches (which indiscriminately diffuse image features over all voxels, causing memory to surge to 3,554MB), QueryAgg's targeted injection strategy achieves a higher mIoUD improvement of +2.35 points with less memory (2,788MB), while also eliminating hallucinated mappings in global attention caused by voxel-image spatial mismatch.
On the RayIoU evaluation (consistency metric along safety rays), StreamOcc achieves 41.1, comprehensively leading other real-time methods across 1m/2m/4m depth thresholds.

Highlights & Insights¶

First successful combination of streaming and dense voxel representations: Prior streaming paradigms were only applied to sparse prediction tasks like object detection and map detection. This work utilizes a dual-aggregation strategy to prove that the high accuracy of dense voxels and the high efficiency of the streaming paradigm can co-exist, opening a new direction for occupancy prediction efficiency optimization.
Exquisitely designed rectification module with dual supervision: Geometry supervision teaches the model "where to attend," while semantic supervision teaches it "what to produce." With clear division of labor, cooperation during training, and zero overhead during inference, this represents a typical training strategy innovation of "teaching the model what to focus on" rather than "feeding more data."
"Less is more" targeted injection design philosophy: QueryAgg only injects features into active dynamic object regions instead of global diffusion. This both avoids hallucinated mappings from image-voxel mismatches and saves computation. This design concept can be transferred to other multimodal feature fusion tasks (e.g., temporal fusion in BEV perception, multi-camera feature aggregation).
Query selection strategy balancing large and small objects: The resolution-aware filtering, using IoU for large targets and geometric constraints for small targets, prevents a single metric from failing on distant small targets, demonstrating solid engineering execution.

Limitations & Future Work¶

It relies on an independent detector (Sparse4Dv3) to generate instance queries, which increases system complexity and requires 3D bounding box annotations during training, making it unable to train independently on datasets with only semantic occupancy annotations.
The two-stage design (StreamAgg followed by QueryAgg) limits the potential for end-to-end joint optimization, as the interaction between the two aggregation modules is a one-way feature transfer.
Experiments were conducted only on nuScenes and its derivative datasets, without validating generalization under more diverse scenarios (e.g., night-time, rain/snow, or different sensor configurations).
The voxel grid resolution is fixed at 0.4m/0.5m; modeling capabilities for extremely small distant objects remain limited by discretization granularity.

vs. Dense multi-frame methods such as COTR/PanoOcc: COTR uses multi-frame concatenation with spatial cross-attention for dense voxel fusion, achieving high precision but with a latency of over 1s and memory exceeding 10GB, totally making deployment unfeasible. StreamOcc reduces computational overhead by an order of magnitude using a streaming paradigm, taking a substantial step forward on the efficiency-precision Pareto frontier.
vs. Sparse streaming methods such as GaussianWorld/ViewFormer: These methods compress scenes into Gaussian primitives or BEV features to adapt to the streaming paradigm, but compression naturally loses dense 3D details. StreamOcc proves that streaming and dense representations are not mutually exclusive, provided that the dual bottlenecks of warping and dynamic objects are addressed.
vs. Real-time methods such as ALOcc-mini: ALOcc-mini achieves 40.6 mIoU with an ultra-low latency of 33ms, but scores only 35.6 on dynamic targets. Although StreamOcc has a slightly higher latency (83ms), it widens the gap by 2.5 points on dynamic targets, which is highly significant for autonomous driving safety.
vs. Global cross-attention image-voxel fusion: Traditional approaches (e.g., COTR, OccFormer, GEOcc) indiscriminately distribute image features to all voxels via spatial cross-attention, which is computationally expensive and prone to hallucination. QueryAgg's targeted injection strategy focusing only on dynamic objects is far more efficient and accurate.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Successfully integrates the streaming paradigm with dense voxel representations for 3D occupancy prediction for the first time. The dual targeted aggregation strategies are highly insightful.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated against 10+ methods across two benchmarks, with ablation studies thoroughly covering each module and design choice without omission.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is clear, the method explanation is highly structured, and the ablation study progresses logically with abundant details supplied in the supplementary materials.
Value: ⭐⭐⭐⭐⭐ Achieves a substantial breakthrough in the accuracy-efficiency trade-off, shifting dense voxel occupancy prediction from unfeasible to near-real-time deployment, driving occupancy prediction closer to practical application.