Skip to content

RECO: Region-Aware Compensation for Extrinsic Perturbations in Roadside 3D Detection

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Autonomous Driving
Keywords: Roadside 3D Detection, Extrinsic Perturbation Compensation, Region-Aware Pose Estimation, Differentiable Soft Gating, Vehicle-to-Infrastructure

TL;DR

To tackle severe feature misalignment caused by roadside camera vibrations and structural deformations, RECO predicts a learnable near/far range boundary alongside piecewise 6-DoF pose offsets to compensate for nominal extrinsics, smoothly blending geometries via a differentiable sigmoid gate and auxiliary reprojection loss to achieve superior 3D detection robustness under both transient and persistent extrinsic perturbations.

Background & Motivation

In vehicle-to-infrastructure (V2I) cooperative systems and intelligent transportation networks, roadside monocular 3D object detection leverages elevated vantage points to cover broad intersection scenes and blind zones well beyond vehicle-mounted sensor horizons, forming the cornerstone for traffic management and proactive collision warnings. However, existing bird's-eye-view (BEV) roadside detectors strictly depend on accurate and static camera extrinsic parameters. Under perspective ground projection, slight camera pose deviations are dramatically amplified along long-range projection rays, resulting in severe image-to-BEV feature misalignment, spatial distortion, and catastrophic localization failure at distant ranges.

Facing extrinsic calibration uncertainties, prior literature generally bifurcates into two paradigms, both exhibiting fundamental limitations. One line of work treats extrinsics as fixed, attempting to stabilize BEV representations through height-guided lifting (e.g., BEVHeight) or smoother feature aggregation; yet their projection operators still rely on corrupted nominal extrinsics. Conversely, calibration-free approaches either bypass explicit calibration via implicit image-to-BEV mappings or rely on external V2X communication anchors. These methods sacrifice geometric metric consistency and introduce fragile dependencies on communication bandwidth and anchor reliability. Crucially, roadside extrinsic perturbations exhibit strong range-dependent error heterogeneity: near-range misalignment is dominated by translation bias, whereas far-range projection is extraordinarily sensitive to rotational jitter. Furthermore, supervision signals across depth ranges are inherently imbalanced. Regressing a single global 6-DoF rigid correction implicitly assumes uniform spatial errors and easily suffers from optimization collapse dominated by specific distance intervals.

This paper approaches the problem by modeling extrinsic refinement not as a single global rigid transformation, but as a range-adaptive piecewise compensation. Core idea: predict a learnable near/far spatial boundary and piecewise 6-DoF pose offsets, smoothly interpolating the compensated geometries via a differentiable soft gate under auxiliary reprojection supervision to eliminate projection misalignment directly within the BEV feature lifting stage.

Method

Overall Architecture

The overall RECO architecture takes a single roadside monocular RGB image alongside its nominal camera extrinsic matrix and outputs robust 3D bounding box predictions. The input image is first processed by a ResNet-50 backbone paired with LSS-FPN to extract multi-scale 2D visual features. The visual features and nominal extrinsics are concatenated and encoded by a Region-Aware (RA) module to jointly predict a dynamic spatial boundary and two sets of 6-DoF pose offsets corresponding to near and far regions. Next, the Soft Compensation Projection (SCP) module converts the offsets into corrected transforms and blends the resulting geometry fields using a differentiable sigmoid gate, establishing a continuous sampling field to scatter-pool perspective features onto a discrete BEV grid. Finally, a standard BEV detection head performs multi-scale dense 3D bounding box regression and classification.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Roadside Image + Nominal Extrinsics"] --> B["ResNet-50 + LSS-FPN<br/>Multi-Scale Visual Feature Extraction"]
    B --> C["Region-Aware Module<br/>Predict Boundary & Near/Far 6-DoF Offsets"]
    C --> D["Soft Compensation Projection Module<br/>Sigmoid Gated Continuous Geometry Blending"]
    D --> E["Scatter-Based BEV Pooling<br/>Construct Smooth Metric BEV Feature Map"]
    E --> F["BEV 3D Detection Head<br/>Dense Classification & 3D Bounding Box Regression"]
    F --> G["Final 3D Bounding Box Predictions"]

Key Designs

1. Region-aware piecewise pose prediction: Decoupling heterogeneous translation and rotation misalignment

In roadside perception, near-range pixel discrepancy is largely governed by camera 3D translations, whereas at distances beyond tens of meters, even fractional-degree rotational vibrations shift ground back-projections by several meters. A single global 6-DoF correction fails under spatially non-uniform supervision signals. To overcome this, the Region-Aware module concatenates aggregated image features \(H_I \in \mathbb{R}^{d \times H_{img} \times W_{img}}\) with the nominal extrinsic matrix \(T \in \mathbb{R}^{4 \times 4}\) into a shared latent vector \(Z\). A boundary head predicts a spatial boundary \(b\) through a learned residual: $\(b = \tilde{b} + \tanh(U_b \cdot Z)\)$ where \(\tilde{b}\) is a prior distance partition (e.g., 20 m or 40 m) and \(U_b\) is a learnable weight matrix. Concurrently, a pose regression head outputs two sets of 6-DoF offsets \(\Delta p_n, \Delta p_f \in \mathbb{R}^{1 \times 6}\) across roll, pitch, yaw, and \(t_x, t_y, t_z\). This design tailors pose refinement specifically to the distinct error characteristics of near and far ranges.

2. Differentiable soft gating: Eliminating feature discontinuities at partition boundaries

Applying hard piecewise extrinsics introduces an abrupt spatial discontinuity at boundary \(b\), causing artificial sampling gaps, duplicate artifacts, and broken gradient propagation across BEV cells. The Soft Compensation Projection module converts the predicted offsets into \(SE(3)\) correction transforms, left-multiplying them with nominal extrinsics to obtain corrected projections \(P_n = \Delta T_n T\) and \(P_f = \Delta T_f T\). For each BEV cell coordinate \((x, y)\), its radial distance \(r = \sqrt{x^2 + y^2}\) is evaluated through a temperature-controlled sigmoid gate: $\(g = \sigma\left(\frac{r - b}{\tau}\right) = \frac{1}{1 + \exp\left(-\frac{r - b}{\tau}\right)}\)$ The composite continuous geometry field \(G''\) is obtained by convex interpolation: $\(G'' = (1 - g) G_n + g G_f\)$ This differentiable blending guarantees mathematical \(C^0\) continuity across the entire BEV plane, completely smoothing out boundary artifacts and providing well-behaved gradient flows for end-to-end back-propagation.

3. Auxiliary reprojection joint supervision: Providing direct geometric gradients for pose refinement

Relying solely on task-level 3D detection loss \(\mathcal{L}_{det}\) offers weak and indirect guidance for unconstrained 6-DoF extrinsic offsets, often leading to sub-optimal local minima. RECO introduces an auxiliary geometric reprojection loss during training. Using the compensated projection operator, 3D ground-truth bounding box corners are back-projected onto the 2D image plane to generate reprojected 2D boxes \(\hat{\mathcal{B}}_{2D}\). An \(L_2\) loss is computed against ground-truth 2D bounding boxes \(\mathcal{B}_{2D}\) over valid instances: $\(\mathcal{L}_{rep} = \frac{1}{\sum m} \sum m \|\hat{\mathcal{B}}_{2D} - \mathcal{B}_{2D}\|_2^2\)$ where \(m\) denotes the valid instance mask. The joint objective is \(\mathcal{L} = \mathcal{L}_{det} + \lambda_{rep} \mathcal{L}_{rep}\). At inference time, the reprojection branch is discarded, incurring zero computational latency while having endowed the pose estimator with strict geometric consistency during training.

Loss & Training

The network is optimized end-to-end with the combined loss \(\mathcal{L} = \mathcal{L}_{det} + \lambda_{rep} \mathcal{L}_{rep}\) using \(\lambda_{rep} = 0.1\) and gating temperature \(\tau = 0.5\). Training employs the AdamW optimizer with a weight decay of \(10^{-4}\). The BEV grid is configured spanning \([-51.2\,\text{m}, 51.2\,\text{m}]\) laterally and \([0\,\text{m}, 140\,\text{m}]\) longitudinally. Input image resolution is fixed at \(864 \times 1536\). Data augmentation encompasses image-level random cropping/rotation and BEV-level scaling/flipping. Models are trained across 4 NVIDIA RTX A6000 GPUs with a batch size of 8 for 50 epochs.

Key Experimental Results

Main Results

On the DAIR-V2X-I validation set, models are evaluated under zero-mean Gaussian transient perturbations \(\mathcal{N}(0, 0.5)\) along yaw and z-axis directions (Car IoU threshold 0.5, Pedestrian and Cyclist IoU threshold 0.25):

Method Deviation Car (Easy) Car (Mod.) Car (Hard) Ped. (Mod.) Cyc. (Mod.) Inference FPS
BEVHeight (CVPR 2023) yaw 65.56 61.79 61.88 10.36 38.21 19.81
CoBEV (TIP 2024) yaw 31.43 26.60 26.97 5.04 22.56 10.16
BEVSpread (CVPR 2024) yaw 67.10 56.04 56.12 15.34 41.20 7.78
HeightFormer (TGRS 2024) yaw 38.64 32.08 32.15 8.90 14.74 8.66
BEVHeight++ (TPAMI 2025) yaw 69.69 58.30 60.38 5.21 18.79 7.80
RECO (Ours) yaw 70.46 63.01 63.03 15.38 42.90 17.66
BEVHeight (CVPR 2023) z-axis 58.57 52.87 53.09 3.30 25.86 19.82
CoBEV (TIP 2024) z-axis 26.80 22.90 22.86 6.27 26.58 10.16
BEVSpread (CVPR 2024) z-axis 53.93 46.48 46.24 10.88 27.15 10.17
HeightFormer (TGRS 2024) z-axis 27.64 20.08 20.00 8.90 14.74 8.67
BEVHeight++ (TPAMI 2025) z-axis 55.80 47.17 47.15 8.60 31.81 7.79
RECO (Ours) z-axis 72.62 64.08 64.17 12.52 35.33 17.66

On the Rope3D benchmark, RECO demonstrates substantial improvements: under yaw jitter, Car Moderate AP reaches 62.34% (outperforming BEVHeight++ at 47.43% and BEVHeight at 39.15%) and Big Vehicle reaches 56.25%; under z-axis perturbation, Car Moderate AP reaches 60.41% (a +25.16% margin over BEVHeight++).

Ablation Study

Models trained under transient noise \(\mathcal{N}(0, 0.5)\) are evaluated under persistent mean-shifted extrinsic deviations \(\mu \in \{-2, -1, 0, 1, 2\}\) on DAIR-V2X-I (Car Easy AP, IoU=0.5):

Config Deviation \(\mu = -2\) \(\mu = -1\) \(\mu = 0\) \(\mu = +1\) \(\mu = +2\) Analysis & Findings
No_Comp (Baseline) yaw 18.03 56.77 67.56 51.45 10.30 Severe degradation under mean shifts; no tolerance
Global_Comp (Single 6-DoF) yaw 32.26 33.25 33.24 33.24 33.04 Optimization plateau; trapped by conflicting gradients
Hard_Comp (Hard Boundary) yaw 40.08 64.93 68.05 65.76 40.10 Range awareness helps, but boundary is discontinuous
Soft_Comp (RECO Full) yaw 49.34 70.08 70.46 70.11 44.36 Smooth soft gating delivers optimal generalization
No_Comp (Baseline) z-axis 5.31 16.80 48.33 16.20 4.66 Ground plane projection completely misaligned
Global_Comp (Single 6-DoF) z-axis 23.00 23.27 23.27 23.31 23.27 Unable to capture coupled pitch and scale changes
Hard_Comp (Hard Boundary) z-axis 31.62 36.89 68.32 65.09 56.75 Substantial gains, but drops at extreme negative shift
Soft_Comp (RECO Full) z-axis 31.10 37.38 72.62 64.36 57.28 Soft gating maintains robust metric geometry across shifts

Key Findings

  • Piecewise range-adaptive compensation is decisive over single global compensation: a single global 6-DoF correction stagnates at roughly 33% AP due to conflicting optimization gradients across distance intervals, whereas region-aware modeling decouples near/far error modes and restores performance to above 70%.
  • Differentiable soft gating eliminates boundary discontinuities and improves optimization stability: under extreme shifts (e.g., yaw \(\pm 2^\circ\)), Soft_Comp outperforms Hard_Comp by 9.26% and 4.26% AP.
  • Perturbation sensitivity shows an asymmetric pattern: yaw rotation exhibits symmetric left-right tolerance, whereas z-axis translation displays strong directional asymmetry, tolerating positive shifts (camera elevation) much better than negative shifts (camera lowering) due to perspective scale compression effects.

Highlights & Insights

  • Online extrinsic compensation embedded directly within geometric lifting: unlike detached pre-calibration networks, RECO drives pose correction via downstream 3D detection and reprojection losses, achieving task-driven spatial alignment.
  • Range-decoupled formulation with differentiable relaxation: addressing the non-linear divergence of projection errors with distance, the framework pairs adaptive range boundaries with continuous sigmoid blending, preserving both spatial flexibility and gradient smoothness.
  • Zero-cost geometric guidance at inference: using 3D-to-2D bounding box reprojection during training enforces rigorous spatial consistency, while discarding this auxiliary branch at test time preserves high real-time efficiency (17.66 FPS).

Limitations & Future Work

  • The current evaluation focuses on single-camera roadside perception and has not yet addressed joint multi-camera calibration consistency across overlapping infrastructure viewpoints.
  • The spatial partitioning is configured with a two-region division (\(k=2\)); extremely deep scenes spanning hundreds of meters may benefit from multi-interval or continuous parameter fields.
  • Extending this region-aware compensation mechanism to vehicle-to-infrastructure (V2I) cooperative networks represents an exciting future direction for global cross-agent calibration.
  • vs BEVHeight / BEVHeight++: BEVHeight reformulates depth into height estimation to enhance tolerance against vertical errors, but strictly presumes fixed nominal extrinsics. RECO actively corrects dynamic extrinsic drift, achieving a 10% to 25% AP margin over BEVHeight++ under perturbations.
  • vs CBR (Calibration-free BEV): CBR foregoes camera extrinsics entirely using implicit representations, which degrades metric scale consistency and creates brittle dependence on external anchors. RECO retains the metric projection framework while estimating residual offsets, achieving both geometric interpretability and high perturbation tolerance.
  • vs LCCNet / CalibFormer: Traditional sensor calibration approaches require heavy cross-modal LiDAR-camera matching, rendering them unsuitable for low-cost monocular roadside units. RECO enables self-contained, end-to-end extrinsic compensation from monocular vision alone.

Rating

  • Novelty: ⭐⭐⭐⭐ [Clever decoupling of range-dependent extrinsic error characteristics using dynamic boundaries and soft-gated compensation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluations across transient noise, persistent drift, pitch/roll generalization, and BEV displacement vector fields]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, solid geometric motivation, and coherent empirical exposition]
  • Value: ⭐⭐⭐⭐⭐ [Directly resolves the persistent real-world challenge of roadside sensor vibration and thermal drift in practical V2X deployments]