FlashBEV: Fast and Memory-Efficient Exact BEV Transformation with IO-Awareness

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yokosyun/FlashBEV
Area: VLM Efficiency
Keywords: bird's-eye-view, view transformation, IO-awareness, kernel fusion, memory efficiency

TL;DR

FlashBEV recasts sampling-based view transformation (Sampling-VT) into a gather reduction operator, eliminating large intermediate 3D tensors via a single fused CUDA kernel with thread-local reduction and dynamic recomputation, slashing peak memory by over 37× and achieving 5.2× speedup while preserving exact mathematical equivalence.

Background & Motivation

Bird's-eye-view (BEV) perception serves as the foundational multi-sensor representation for 3D scene understanding in autonomous driving systems. At the center of every camera-based BEV pipeline lies view transformation (VT), which maps 2D multi-camera image features into a unified 3D ego-coordinate frame. Existing view transformation pipelines generally fall into two broad categories: splatting-based VT (Splatting-VT), typified by Lift-Splat-Shoot (LSS) and BEVPoolv2, which forward-projects 2D features into 3D voxel space based on predicted depth distributions; and sampling-based VT (Sampling-VT), represented by SimpleBEV and related architectures, which backward-queries multi-camera feature maps from dense 3D coordinate grids. While splatting methods are often memory-friendly via index-based pooling, they typically produce non-uniform and geometry-dependent scene coverage. Conversely, Sampling-VT offers dense, continuous, and robust feature aggregation, making it particularly appealing for long-range and high-resolution perception.

However, deploying Sampling-VT on practical automotive compute platforms has long been hindered by severe system-level memory bottlenecks. In standard implementations—which the authors term Tensorized Sampling-VT—the transformation relies on discrete, tensorized PyTorch operations that sequentially materialize enormous 3D feature tensors spanning BEV grid dimensions, height discretizations, and camera views. Under typical benchmark configurations (e.g., 50 m perception range, 200×200 grid resolution, 8 vertical bins, and batch size 1), the view transformation step alone consumes nearly 2 GB of peak GPU memory. When scaling up spatial resolution or vertical discretization to capture fine 3D structures, intermediate memory traffic scales poorly as \(O(BNCXYZ)\), saturating high-bandwidth memory (HBM) and causing out-of-memory (OOM) failures. Prior efforts to mitigate this burden have either traded away dense coverage via sparse sampling or sacrificed continuous bilinear accuracy using quantized offline lookup tables (LUTs).

The central insight of this paper is that the scalability bottleneck in Sampling-VT does not stem from its underlying mathematical formulation, but entirely from its execution paradigm. Core idea: by recognizing Sampling-VT as an embarrassingly parallel gather reduction operator, FlashBEV executes the entire view transformation within a single IO-aware fused CUDA kernel that maps each output BEV cell to a single GPU thread, performing on-the-fly projection, bilinear interpolation, and register-level reduction without materializing any camera- or height-dependent intermediate tensors.

Method

Overall Architecture

In canonical Sampling-VT, the model takes as input \(N\) multi-camera feature maps \(F_n \in \mathbb{R}^{B \times N \times C \times H \times W}\) and their corresponding camera projection matrices \(P \in \mathbb{R}^{B \times N \times 3 \times 4}\). For each BEV spatial coordinate \((x, y)\) and feature channel \(c\), features are aggregated across \(Z\) discrete vertical height bins by computing the valid mean across cameras for each height bin \(z\) and then summing across all bins \(z\):

\[B(x, y, c) = \sum_{z=1}^{Z} \frac{\sum_{n=1}^{N} M_n(x, y, z) f_n(x, y, z, c)}{\max\left(1, \sum_{n=1}^{N} M_n(x, y, z)\right)}\]

where \(f_n(x, y, z, c)\) denotes the bilinearly sampled feature from camera \(n\)'s feature map at the projected voxel center, and \(M_n(x, y, z) \in \{0, 1\}\) is a field-of-view (FoV) validity indicator that equals 1 when the 3D voxel center projects inside the image plane with positive depth, and 0 otherwise.

Traditional tensorized execution splits this mathematical formula into multiple independent steps: voxel grid generation, voxel-to-camera projection, grid sampling, validity masking, cross-camera reduction, and vertical summation. Each step materializes and writes full-sized high-dimensional tensors back to global memory (HBM). FlashBEV capitalizes on the observation that the gather reduction pattern exhibits zero cross-spatial dependencies across distinct BEV grid positions \((x, y)\), thereby collapsing the entire sequence into a single fused GPU kernel.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-camera features & projection matrices<br/>Fn (B,N,C,H,W) / P (B,N,3,4)"] --> B["Thread-local mapping and zero-intermediate reduction<br/>one GPU thread bound to a single (b,x,y,c)"]
    B --> C["Register-level projection and bilinear sampling<br/>on-the-fly coordinate transforms and memory lookups"]
    D --> E["Final BEV feature output<br/>B (B,C,X,Y) / strictly O(BCXY) memory"]
    C --> D["Per-height local normalization & single-write output<br/>register accumulation with one final HBM write"]

Key Designs

1. Thread-local mapping and zero-intermediate reduction: breaking the tensor materialization bottleneck

Standard tensorized pipelines execute operators across full tensors, allocating intermediate tensors of shape \((B, X, Y, Z, 3)\) for world grids, \((B, N, X, Y, Z, 3)\) for projected coordinates, and \((B, N, C, X, Y, Z)\) for sampled features. This massive allocation overwhelms global memory bandwidth. FlashBEV completely bypasses tensor materialization by reordering execution: it assigns each GPU thread to a single unique output element \((b, x, y, c)\). Because each output pillar aggregates only the specific rays corresponding to its own spatial coordinate, threads execute completely independently without inter-thread communication, shared-memory synchronizations, or atomic locks. All intermediate coordinates and features are kept purely in registers, bounding operator peak memory strictly to the output feature map size \(O(BCXY)\) and making memory consumption entirely invariant to the number of height bins \(Z\) and cameras \(N\).

2. Register-level projection and bilinear sampling: trading arithmetic compute for memory bandwidth

In standard deep learning compilation, geometric projections are computed once and written to HBM to avoid duplicate arithmetic across feature channels. However, on modern GPU architectures, arithmetic throughput vastly outpaces global memory bandwidth. FlashBEV embraces an aggressive "recomputation over memory traffic" philosophy: while 3D voxel coordinates and their 2D camera projections are independent of channel \(c\), FlashBEV recomputes perspective projection, coordinate scaling, and boundary checks on-the-fly inside each thread's private registers. The thread directly fetches the four nearest pixel values from image feature memory to perform bilinear interpolation. Intermediate projected coordinates exist for only a few clock cycles within registers before being consumed and overwritten, completely eliminating gigabytes of intermediate grid read/write traffic.

3. Per-height local normalization and single-write output: exact mathematical equivalence in registers

To avoid brightness and magnitude distortion in multi-camera overlap zones, valid features must be normalized by the count of visible cameras per height bin before vertical aggregation. FlashBEV manages this inside a compact two-level nested loop within each thread: an outer loop iterating over height bins \(z \in \{1, \dots, Z\}\) and an inner loop iterating over camera indices \(n \in \{1, \dots, N\}\). For each height bin \(z\), the thread accumulates valid features \(\text{num}_z \leftarrow \text{num}_z + M_n f_n\) and increments the validity counter \(\text{den}_z \leftarrow \text{den}_z + M_n\). Upon finishing the inner loop, it computes the local normalized mean \(g_z = \text{num}_z / \max(1, \text{den}_z)\) and accumulates it into an overall accumulator register \(\text{acc} \leftarrow \text{acc} + g_z\). Only after traversing all \(Z\) vertical bins does the thread write the accumulated scalar \(\text{acc}\) to global memory once. This guarantees strict mathematical equivalence to Equation (1), keeping forward and backward numerical discrepancies strictly within single-precision floating-point round-off tolerances (maximum forward absolute error of \(2.93 \times 10^{-4}\) and mean error of \(8.79 \times 10^{-6}\)).

A Worked Example

Consider processing a single scene (\(B=1\)) with a standard 6-camera surround-view rig (\(N=6\)), a BEV grid of \(200 \times 200\), \(Z=8\) vertical height bins, and \(C=128\) feature channels. 1. Tensorized Baseline Execution: The framework first allocates and writes the \((1, 200, 200, 8, 3)\) coordinate grid to HBM, then projects it to 6 cameras to generate a \((1, 6, 200, 200, 8, 3)\) grid. It calls grid_sample to materialize a \((1, 6, 128, 200, 200, 8)\) FP32 feature tensor (consuming \(\approx 983 \text{ MB}\) for this single tensor alone), followed by masking and reduction tensors. In total, the baseline performs dozens of separate kernel launches and reaches \(1971.9 \text{ MB}\) of peak memory. 2. FlashBEV Fused Execution: The runtime launches \(1 \times 128 \times 200 \times 200 = 5,120,000\) CUDA threads. The thread assigned to \((x=100, y=100, c=32)\) initializes its local register acc to 0. It iterates through \(z=1 \dots 8\): at \(z=1\), it resets num and den to 0, loops through cameras \(1 \dots 6\), finds that the coordinate projects validly into cameras 1 and 2, bilinearly samples their features \(f_1, f_2\), updates \(\text{num} = f_1 + f_2, \text{den} = 2\), computes \(g_1 = (f_1 + f_2) / 2\), and adds it to acc. After completing all 8 height iterations, the thread performs a single global write to \(B(0, 32, 100, 100)\). Peak operator memory is restricted to the \(52.8 \text{ MB}\) output tensor, requiring zero intermediate allocations.

Key Experimental Results

Main Results

On an NVIDIA RTX A6000 GPU under the standard SimpleBEV reference setup (\(X=Y=200, Z=8, C=128, B=1\)), FlashBEV was benchmarked against the Tensorized Sampling-VT baseline in terms of isolated VT latency, peak memory, end-to-end performance, and downstream 3D detection on the nuScenes validation split.

Module / Scope	Metric	FlashBEV (Ours)	Tensorized Sampling-VT	Gain / Reduction
Isolated VT Operator (Forward)	Latency (ms)	1.77 ± 0.03	9.18 ± 0.23	5.19× speedup (-80.7% time)
Isolated VT Operator (Forward)	Peak Memory (MB)	52.82	1971.94	37.33× reduction (-97.3% memory)
Isolated VT Operator (Backward)	Latency (ms)	3.27 ± 0.07	17.90 ± 0.20	5.47× speedup (-81.7% time)
Isolated VT Operator (Backward)	Peak Memory (MB)	105.63	1174.86	11.12× reduction (-91.0% memory)
End-to-End Perception Model	E2E Latency (ms)	59.91	67.19	10.8% speedup (-7.28 ms)
End-to-End Perception Model	E2E Peak Memory (MB)	876.76	2211.29	2.52× reduction (-60.3% memory)
Downstream Detection (nuScenes)	3D IoU (%)	46.8 ± 0.1	46.9 ± 0.3	Numerically equivalent (within variance)

Ablation Study

To investigate the scalability of FlashBEV across vertical discretization resolutions and its performance within production-grade deployment engines, the authors implemented FlashBEV as a TensorRT plugin and evaluated it on an NVIDIA RTX 4070 Ti SUPER GPU (FP32 precision, grid \(200 \times 200, C=128\)):

Height Bins \(Z\)	Implementation	Peak Memory (MB)	Memory Comp. Ratio	Latency (ms)	Speedup Ratio	Max Absolute Error
\(Z = 8\)	TensorRT Baseline	1294	1.0×	5.622	1.0×	0.0
\(Z = 8\)	FlashBEV TRT Plugin	38	34.1× reduction	1.128	5.0× speedup	0.0 (exact)
\(Z = 16\)	TensorRT Baseline	2514	1.0×	20.132	1.0×	0.0
\(Z = 16\)	FlashBEV TRT Plugin	38	66.2× reduction	2.140	9.4× speedup	0.0 (exact)
\(Z = 32\)	TensorRT Baseline	4994	1.0×	40.193	1.0×	0.0
\(Z = 32\)	FlashBEV TRT Plugin	38	131.4× reduction	4.177	9.6× speedup	0.0 (exact)

When extended to BEVFormer's spatial cross-attention (SCA) as FlashSCA sweeping sampling points \(P\) per ray, at \(P=32\) FlashSCA compressed peak memory from 3708 MB to 1168 MB (3.18× reduction) and cut latency from 29.27 ms to 13.90 ms (2.11× speedup).

Key Findings

• Complete Decoupling of Memory from Vertical Bins: Peak memory in Tensorized Sampling-VT grows strictly linearly with \(Z\), exceeding 4.9 GB at \(Z=32\). FlashBEV remains flat at 38 MB in TensorRT regardless of \(Z\), enabling ultra-fine vertical discretization without additional memory cost.
• Order-of-Magnitude Grid Scaling: Under the baseline's memory ceiling (1972 MB), FlashBEV expands the maximum attainable BEV grid from \(200 \times 200\) to \(1992 \times 1992\) (a ~10× increase in spatial dimension).
• Universal Hardware Speedups: Across six heterogeneous GPU architectures spanning enterprise accelerators (H200, A6000, A4000), consumer cards (RTX 4060, RTX 2060), and edge automotive modules (Jetson Orin Nano 8GB), FlashBEV consistently delivers 3.3×–6.2× forward and 5.2×–7.9× backward speedups.

Highlights & Insights

• Exact Operator-Level Equivalence: Unlike approximation techniques relying on pruning or quantization, FlashBEV maintains bit-level algorithmic integrity, achieving dramatic physical speedups without model retraining or performance degradation.
• Recomputation as a Modern Hardware Paradigm: Translating FlashAttention's memory hierarchy insight to 3D autonomous driving, FlashBEV proves that re-evaluating coordinate math in local registers is orders of magnitude cheaper than reading and writing large intermediate feature grids across HBM.
• Seamless Cross-Architecture Portability: The proposed execution model readily transfers to attention-based view transformations (e.g., BEVFormer SCA), demonstrating broad applicability across diverse 3D perception architectures.

Limitations & Future Work

• Redundant Geometry Calculation across Channels: Because projection geometry is identical across channels, assigning independent threads to \((b, x, y, c)\) duplicates coordinate transforms \(C\) times. For wide channel settings (\(C \ge 512\)), this arithmetic overhead narrows the latency advantage, suggesting a potential improvement via warp-level cooperation or shared-memory coordinate caching.
• Single-Scale Input Focus: The current kernel formulation targets single-scale feature inputs; extending the fused gather reduction to multi-scale feature pyramids (e.g., FPN) represents an important next step for high-resolution detection.

Related Work & Insights

• vs. Splatting-VT (LSS / BEVPoolv2): Splatting forward-scatters features based on depth estimation, requiring sorting and suffering from non-uniform spatial distribution; FlashBEV retains the uniform density and backward querying of Sampling-VT while removing its historical memory penalty.
• vs. Lookup Table Methods (FastBEV / DualBEV): FastBEV eliminates runtime projection via static offline tables but limits voxels to single-camera sampling and introduces discretization error; FlashBEV requires no offline precomputation and performs exact continuous bilinear interpolation.
• vs. Sparse Sampling (PointBeV / SparseBEV): Sparse methods reduce overhead by skipping background regions; FlashBEV solves the foundational memory wall at the low-level kernel execution layer, enabling scalable, fully dense BEV representations.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant identification of the gather reduction pattern in Sampling-VT, applying IO-aware principles to autonomous driving operators.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-GPU benchmarking (6 platforms), TensorRT deployment verification, and architectural transfer to BEVFormer.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous systems analysis, lucid mathematical exposition, and crystal-clear visual and tabular presentations.
• Value: ⭐⭐⭐⭐⭐ High industrial and academic utility, providing an immediate drop-in replacement that enables scalable, fine-grained BEV perception on memory-constrained hardware.