RadarGen: Automotive Radar Point Cloud Generation from Cameras

Conference: ECCV2026
Paper: ECCV Paper
Project: https://radargen.github.io/
Area: Autonomous Driving
Keywords: radar point cloud generation, bird's-eye view, latent diffusion, radar cross section, Doppler velocity

TL;DR

RadarGen extracts BEV visual conditions from surround cameras at two timestamps, jointly generates radar density, RCS, and Doppler maps, and recovers sparse points, improving foreground box hit rate from 0.37 to 0.66 on MAN TruckScenes while remaining substantially below real radar in downstream detection quality.

Background & Motivation

Autonomous-driving simulation can already generate or edit convincing images, but visual content alone cannot test perception systems that depend on radar. Radar supplies more than another surface geometry measurement: each detection carries radar cross section (RCS), reflecting scattering strength, and Doppler velocity, measuring relative radial motion. Materials, incidence angles, multipath, and interference change returns, so the same visual scene can admit different plausible radar observations. A deterministic image-to-point mapping can underrepresent this variability, while directly transferring a LiDAR generator creates a representation mismatch. LiDAR can often be organized into relatively dense range images using scanning angles, whereas peak-detected radar points are sparse and nonuniform and do not fit that structure naturally.

An alternative is electromagnetic radar simulation, but it requires scene assets, materials, and sensor specifications, making long-tail coverage expensive to engineer. Recorded datasets also commonly retain only point clouds after range-Doppler transforms, beamforming, and detection, having discarded raw waveforms and phase information. These processing chains are often proprietary, making it impractical to reconstruct a complete physical sensor from the available data alone. RadarGen therefore targets the processed point clouds available in practical datasets rather than raw radar signals. Camera conditioning provides an interface to existing visual data, editing tools, and simulators, but does not make all invisible scattering factors identifiable from images.

The paper reuses SANA's image-latent diffusion capabilities by putting both radar targets and visual conditions into spatially aligned bird's-eye-view (BEV) maps. Density smoothing makes sparse points easier to encode, while depth, semantic, and motion priors reduce the need for the generator to relearn scene structure from pixels. This choice also retains an explicit inverse process: after image decoding, discrete detections must be recovered rather than treating a blurred heatmap as a point cloud. Core Idea: represent radar through recoverable BEV maps, constrain stochastic generation with aligned visual priors, and use sparse deconvolution to turn continuous predictions back into radar points with attributes.

Method

Overall Architecture

Inputs are surround-camera images at \(t\) and \(t+\Delta t\), together with known camera intrinsics and extrinsics. The output is a set of radar detections at \(t\) in the ego-vehicle frame, with planar coordinates, RCS, and Doppler velocity, rather than full elevation or raw waveforms. Although the paper describes point cloud generation, this implementation produces an attributed planar point set; elevation is discarded when constructing BEV targets. During training, real radar supplies supervision through the Recoverable BEV Representation, while cameras supply BEV Visual Conditioning; the branches meet at Joint Latent Denoising. Inference requires no real radar, starts from noise to generate three maps, and uses Sparse Deconvolution Recovery to obtain discrete points and attributes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Radar["Real radar<br/>training only"] --> Maps["Recoverable BEV<br/>Representation"]
    Images["Two-timestamp surround images<br/>camera calibration"] --> Cues["BEV Visual<br/>Conditioning"]
    Maps -.->|Encoded noisy supervision| Joint["Joint Latent<br/>Denoising"]
    Cues -->|Spatially aligned conditions| Joint
    Noise["Gaussian noise<br/>inference start"] --> Joint
    Joint -->|Decode three radar maps| Recover["Sparse Deconvolution<br/>Recovery"]
    Recover --> Points["Planar coordinates + RCS<br/>Doppler velocity"]

The real-radar branch supplies training targets only; it is not an additional sensor input that must be collected at inference time. The two timestamps are genuine conditioning inputs: the later one supports motion estimation rather than requesting generation of the next radar frame. The architecture and method are documented on pp. 6–10, Section 4, and Figures 3–4.

Key Designs

1. Recoverable BEV Representation: put sparse radar into an existing image latent space

The method first discards elevation and rasterizes detections into a sparse planar point map, motivated by the limited vertical resolution of automotive radar. Isolated bright pixels are difficult for a frozen image autoencoder to reconstruct, so positions are supervised through a point density map obtained by convolution with a fixed Gaussian kernel. This converts sparse observations into a smooth image while retaining a known forward process that can later support deconvolution. The clearly readable relationship in Section 4.1 is:

\[ M_p = K_\sigma * P_{xy}. \]

Here \(P_{xy}\) is the rasterized point map, \(K_\sigma\) is the fixed Gaussian kernel, and \(M_p\) is the smoothed point density map. RCS and Doppler are not left at isolated points either: each pixel inherits the attribute of its nearest radar detection, producing piecewise-constant maps through a Voronoi partition. These maps specify which attribute to retrieve if a point is recovered there; they do not claim that actual radar returns occupy the entire plane. Each single-channel map is replicated across three channels and encoded independently, allowing direct reuse of SANA's RGB autoencoder without changing its input interface. Location, RCS, and Doppler therefore have separate latent representations on the same BEV coordinate grid. Compatibility with image encoding has a cost: excessive smoothing merges local structure, so the lowest image reconstruction error need not yield the best point recovery.

2. BEV Visual Conditioning: put appearance, categories, and motion into radar coordinates first

UnidepthV2 predicts metric depth for camera images; camera calibration supports back-projection and transformation into the ego-vehicle frame, followed by multi-view fusion and rasterization. This provides a shared geometric basis for the three conditioning maps, rather than merely appending depth values and expecting the denoiser to learn spatial alignment itself. The appearance map obtains colors from RGB images, while the semantic map obtains category colors from Mask2Former segmentation using a model trained on Cityscapes. Color-coded semantics replace one-hot vectors to preserve image-like conditioning statistics compatible with the pretrained image encoder. These categories help the model infer object-dependent return properties, but they do not directly measure materials or electromagnetic reflectivity.

The third condition is a radial velocity map: UniFlow establishes pixel correspondences across the two timestamps, and depth at both times back-projects them into spatial positions. The authors approximate motion from position change over the elapsed time, retaining the radial component toward the ego vehicle as a Doppler-like cue. This map is neither radar ground truth nor a velocity label inferred solely from one static image; errors depend jointly on flow, depth, and coordinate handling. Projecting all three cues into BEV gives target locations geometric, categorical, and motion context in advance. The velocity equation on p. 9 has missing characters in the extraction, so this note explains the readable prose without reconstructing an exact equation or unspecified ego-motion compensation details.

3. Joint Latent Denoising: share attention while preserving radar attribute identity

The denoiser learns the joint distribution of three radar latent representations given BEV conditions, rather than independently generating positions, reflectivity, and velocity. Because conditions and targets are spatially aligned, conditioning tensors can be concatenated along feature channels without attention having to discover every correspondence from perspective camera views. Latents from all three radar maps are assembled into one token sequence and share self-attention in a Diffusion Transformer (DiT). Joint processing lets positions and attributes influence one another, so density structure and motion near an object need not be guessed by three noncommunicating models. Learnable modality identifier embeddings distinguish each map, preventing density, RCS, and Doppler from being treated as statistically interchangeable channels.

SANA's autoencoder provides 32-fold spatial compression, and its linear attention reduces long-sequence cost; these efficiencies come from the reused backbone rather than a new attention mechanism introduced here. Inference starts from Gaussian noise, iteratively denoises under a fixed scene condition, and uses the frozen decoder to obtain three BEV radar maps. Stochastic sampling expresses alternative plausible returns; it does not imply explicit multipath equations or a guarantee of physical consistency. The image autoencoder remains frozen during training, while the SANA DiT adapted for conditioning is fine-tuned.

4. Sparse Deconvolution Recovery: retrieve discrete detections from smooth density

Local maxima in a predicted density map do not necessarily correspond one-to-one to original points; peak selection can miss overlapping detections, while random sampling can alter their spatial distribution. The authors exploit the known, fixed Gaussian kernel used for training targets and formulate recovery as nonnegative deconvolution with L1 sparsity regularization. A data-fidelity term asks the recovered point map, blurred by the same kernel, to match the generated density, while sparsity discourages interpreting whole regions as returns. The text specifies an iteratively reweighted L1 (IRL1) scheme with a FISTA solver. Thresholding the recovered point map yields coordinates, after which RCS and Doppler are retrieved at matching positions to assemble the output point set.

This explains why attribute maps may densely fill BEV: position recovery determines whether a point exists, and attributes are read only at recovered locations. It also separates generation error from representation error: even density derived from real points can require a robust inverse process after passing through an autoencoder. Equation (1) on p. 10 has missing operators in the extraction, so a conjecturally repaired LASSO expression is not presented as the authors' exact formula; the explanation follows the readable adjacent prose. Figure 7 supports deconvolution over the tested peak, random, and hybrid recovery alternatives, but does not supply curve values that can be reliably transcribed here.

A Worked Example

Consider the Figure 1 edit replacing a farther car with a closer truck; this walkthrough explains the mechanism rather than adding a quantitative experiment. Depth projection and segmentation of edited views change the object's position, coverage, and category conditions in BEV. The complete pipeline also uses corresponding images at two timestamps for radial motion; one edited illustration cannot establish cross-timestamp consistency. The denoiser produces three radar maps under the changed scene condition, and position recovery determines where discrete returns exist. The paper shows returns near the new truck and removal of returns in the newly occluded region, indicating that editing affects more than a local addition of points. Attribute maps then assign RCS and Doppler to recovered points, and the resulting point cloud can be passed to a detector trained on real radar. This example demonstrates controllability for the illustrated scene, not reliability across all occlusions or edited sequences.

Loss & Training

Supervision comes from three BEV maps constructed from real radar, encoded and noised for conditional joint denoising; deconvolution is an inference recovery step, not a detection-training loss. The main paper does not fully specify the diffusion loss, noise schedule, or sampling step count, so no concrete training equation is invented here. Each conditioning input is dropped with 10% probability during training, exposing the model to missing cues. Radar is cropped to ±50 m around the ego vehicle using a 512 × 512 BEV grid; analysis in Figure 7 selects the Gaussian bandwidth parameter \(\sigma=2\). The tested bandwidths are \(\{0.5,1,1.5,2,2.5,3\}\); increasing bandwidth reduces autoencoder reconstruction error, but excessive smoothing harms point recovery. Training takes 2 days on 8 L40 GPUs with 48 GB each, and the model has 592M parameters. These are training resources, not per-frame end-to-end inference latency; the main paper is insufficient to determine onboard real-time deployment speed.

Key Experimental Results

Main Results

The main evaluation uses MAN TruckScenes: after excluding almost invisible dark clips, training and evaluation use 431 and 49 clips, respectively. Each clip contains approximately 200 frames, around 40 with bounding-box annotations; training uses all training frames, while evaluation uses annotated frames only. Table 1 aggregates Entire Area metrics over timestamps and Foreground metrics over objects; the reported mean ± standard deviation is not a confidence interval. The baseline extends RGB2Point to multi-view inputs and RCS/Doppler outputs, has 432M parameters, and uses only time \(t\); RadarGen uses two timestamps, so inputs and capacity are not strictly matched.

The following selection from Table 1 on p. 10 captures overall geometry, attributes, and foreground coverage.

Evaluation scope / metric	RGB2Point baseline	RadarGen	Direction
Entire Area CD Loc.	1.84 ± 0.48	1.68 ± 0.39	Lower is better
Entire Area CD Full	0.038 ± 0.009	0.040 ± 0.008	Lower is better
Entire Area IoU@1m	0.23 ± 0.10	0.31 ± 0.11	Higher is better
Entire Area DA F1	0.14 ± 0.09	0.24 ± 0.12	Higher is better
Entire Area MMD Loc.	0.368 ± 0.151	0.056 ± 0.062	Lower is better
Entire Area MMD RCS	0.36 ± 0.25	0.09 ± 0.15	Lower is better
Entire Area MMD Doppler	0.65 ± 0.64	0.31 ± 0.74	Lower is better
Foreground CD Loc.	1.32 ± 0.79	0.95 ± 0.65	Lower is better
Foreground Density Similarity	0.35 ± 0.43	0.51 ± 0.41	Higher is better
Foreground Hit Rate	0.37	0.66	Higher is better

CD Loc. is position Chamfer distance; CD Full uses normalized location, RCS, and Doppler vectors, so its magnitude is not directly comparable with CD Loc. Density Similarity measures agreement between generated and real point counts inside boxes; Hit Rate measures coverage of real foreground boxes by generated returns. DA metrics consider spatial proximity together with RCS and Doppler similarity, while MMD compares real and generated distributions. Thresholds, density normalization, matching rules, and MMD kernel parameters for these custom metrics are deferred to supplementary material; the available main-paper cache does not support exact metric reproduction.

Ablation Study

The following selection from Table 2 on p. 13 uses the same MAN TruckScenes evaluation; the text describes conditioning ablations as zeroing the relevant condition. MV Camera Cond. directly inputs multi-view image tokens from two timestamps, with Plücker and modality embeddings, without constructing BEV conditions first. Only metric means are listed here; the full table includes standard deviations, and small differences should not be interpreted as statistically significant findings.

Config	Entire Area CD Loc. ↓	Entire Area MMD RCS ↓	Entire Area MMD Doppler ↓	Foreground Hit Rate ↑
MV Camera Cond.	1.88	0.06	0.24	0.56
Without appearance map	1.71	0.09	0.35	0.65
Without semantic map	1.72	0.12	0.33	0.64
Without velocity map	1.69	0.09	0.34	0.66
RadarGen	1.68	0.09	0.31	0.66

Key Findings

The foreground hit-rate increase from 0.37 to 0.66 means the generator more often places returns near real objects, not that detection accuracy is 0.66. The full model does not win everywhere: Entire Area CD Full is 0.040 versus the baseline's better 0.038; the authors attribute this to the baseline's related training objective. Removing semantic conditioning raises Entire Area MMD RCS from 0.09 to 0.12, supporting category cues for attribute modeling, whereas removing velocity leaves foreground hit rate at 0.66. Direct multi-view conditioning achieves MMD RCS of 0.06 and MMD Doppler of 0.24, better than BEV's 0.09 and 0.31, so BEV's advantages concern geometry and efficiency rather than every attribute distribution. Page 14 reports over 3 times the runtime and 9 training days for the multi-view version, versus 2 training days for BEV; an absolute end-to-end latency cannot be verified here. Section 5.1 on p. 12 trains VoxelNeXt on real radar and evaluates different point clouds: NDS is 0.48 on real data, 0.30 on RadarGen, and nearly zero on the baseline. This tests whether a real-data-trained detector can consume generated data; it is not synthetic-data training followed by real-data testing and does not establish detection gains from augmentation on a real test set.

Highlights & Insights

• Design the representation and inverse process together. Smoothing makes radar compatible with an image autoencoder, while the known kernel constrains recovery; image reconstruction alone is not the final geometric objective.
• Align before generating. Projecting depth, semantics, and motion into BEV reduces the radar generator's burden of learning cross-view correspondence, although alignment quality depends on upstream models.
• Evaluate geometry and attributes jointly. Better distribution distances need not mean better object coverage, and a higher box hit rate does not ensure in-domain performance for a real-data detector.

Limitations & Future Work

• The authors acknowledge dependence on visual foundation models, with difficulties under low light, strong reflections, and camera occlusion; almost invisible dark clips are excluded, so radar's weather resilience does not make this whole pipeline weather-resilient.
• Generation outside camera visibility can provide useful completion or uncontrolled hallucinations; semantically consistent examples do not replace systematic evaluation of false returns in hidden regions.
• Explicit radar mechanisms are not modeled, and elevation is discarded; generating processed detections is not equivalent to simulating complete 4D radar measurements or the raw signal-processing chain.
• Two timestamps condition one radar output, rather than ensuring temporally consistent radar videos; the authors identify video, text conditioning, multiple datasets, and multiple radar configurations as future directions.
• Supplementary material is absent from the available cache, leaving nuScenes results, full detection tables, metric details, and additional implementation parameters unverified; damaged velocity and deconvolution equations are not reconstructed by guesswork.

Related Work & Insights

• Compared with RGB2Point. The extended baseline directly regresses points and attributes, whereas RadarGen models a conditional distribution with BEV priors; timestamp count and capacity also differ, so diffusion alone cannot explain every gain.
• Compared with SANA. The paper reuses its high-compression autoencoder and efficient DiT; the additions center on radar representation, visual conditioning, and point recovery rather than a general image-generation backbone.
• Compared with physical simulation and Radar Fields-style reconstruction. The former relies on physical specifications and assets, and the latter on scene radar observations; RadarGen generates observations for new scenes from cameras without equivalent physical interpretability.
• A testable next direction. This note proposes training a detector with synthetic radar and testing gains on real data, separately measuring night, occlusion, and sensor changes; this is a research suggestion, not an experiment completed in the paper.

Rating

• Novelty: 4/5. Camera-conditioned probabilistic generation, recoverable BEV representations, and joint radar attributes form a coherent modality-specific contribution.
• Experimental Thoroughness: 4/5. Geometry, attributes, distributions, conditioning ablations, and detector compatibility are covered, but matched-input controls and real downstream gains remain limited in the main paper.
• Writing Quality: 4/5. Representation, conditioning, and recovery have clear causal connections, although some metrics and implementation details depend on supplementary material.
• Value: 4/5. The method provides a practical route to radar-enriched visual simulation but does not replace real radar collection or safety validation.