Understanding Cross-Rig Generalization in Automotive Perception: a Multi-Rig Benchmark and Rig Variation Metrics

Conference: ECCV2026
arXiv: 2606.27554
Project Page: https://badertim.github.io/plentiful-carla-camera-rigs/
Code: None
Area: Autonomous Driving
Keywords: Camera Rig, Cross-Sensor Generalization, Benchmark Dataset, Autonomous Driving Perception, 3D Object Detection

TL;DR

This paper constructs PCCR, a simulation benchmark containing 14 systematically designed camera rigs, and proposes two geometric descriptors based on calibration parameters: Rig Variance (RigV) and Rig Contrastive Distance (RigCD). Experiments demonstrate that camera translation discrepancy is the dominant factor in cross-rig generalization performance degradation, and RigCD can predict the ranking of transfer difficulty between different rigs, achieving a Spearman rank correlation of up to 0.80.

Background & Motivation

Vision-based autonomous driving perception systems are typically developed and evaluated under fixed sensor configurations—requiring a specific number of cameras, fixed mounting positions, and orientations during both training and deployment. However, real-world vehicle fleets are highly heterogeneous: packaging constraints, cost targets, and generational updates across different vehicle models lead to significant variations in camera mounting positions, overlap angles, field of view (FOV), and even the number of cameras. This shift from a fixed configuration to a heterogeneous fleet introduces a specific domain gap—termed the cross-rig domain gap—which is unique because only the geometric observation process changes, while the scene content itself remains identical. This means that performance degradation stems purely from changes in perspective distribution, coverage, and redundancy, without confounding factors from traditional domain shifts like illumination, weather, or scene categories.

Why has this problem not been systematically studied before? There are two fundamental reasons. First, existing autonomous driving datasets (e.g., nuScenes, Waymo) employ fixed camera rigs, making cross-rig analysis impossible; cross-dataset evaluations confound rig variations with scene statistical differences, preventing isolated attribution. Second, prior studies either evaluated only a few rig variants (such as Embacher et al.'s modification of two vehicle models) or restricted themselves to viewpoint perturbations of a single rig, failing to cover the full spectrum of the rig design space. The CARLA simulator offers a viable solution: it can arbitrarily modify camera calibration parameters while keeping the scene content completely identical, thereby isolating the true impact of rig geometric changes on perception performance. Furthermore, if the difficulty of cross-rig transfer can be predicted solely from calibration metadata, risks can be quickly assessed prior to deploying a new rig, without consuming extensive resources to train and evaluate models.

Following this line of thought, the core contributions of this paper are two-fold. At the data level, this work constructs the Plentiful CARLA Camera Rigs (PCCR) benchmark, consisting of 14 systematically designed camera rigs. By rendering 115 identical driving scenarios for each rig, it obtains 724,500 images along with complete 3D annotations. At the analytical level, it proposes two geometric descriptors relying solely on calibration metadata—Rig Variance (which measures the internal camera diversity within a single rig) and Rig Contrastive Distance (which quantifies the geometric discrepancy between two rigs)—and calibrates their weights to predict the ranking of cross-rig performance degradation. Core Idea: Segregating rig geometric effects from environmental confounding factors through a scene-consistent, multi-rig simulation benchmark, and demonstrating for the first time that the difficulty of perception transfer between different camera layouts can be predicted with a Spearman correlation of \(\rho=0.80\) using only calibration metadata.

Method

Overall Architecture

The proposed method consists of two parts: the pipeline for constructing the PCCR multi-rig benchmark, and the geometric rig descriptor framework based on calibration metadata. The benchmark construction starts with global metadata configurations (map, weather, traffic density, and rig specifications). After sampling scenario descriptors, it initializes the CARLA simulation environment and records the complete trajectories of all entities. Trajectory-level pruning is then applied to reduce the oversampling of near-static ego-behaviors. Finally, deterministic replay ensures that all 14 rigs observe identical scene dynamics. For calibration descriptors, the camera position, orientation, and FOV of each rig are extracted as metadata to calculate RigV (internal rig diversity) and RigCD (inter-rig geometric discrepancy). The weights of RigCD components are then calibrated using L-BFGS-B on controlled rig variants to maximize the Spearman rank correlation between RigCD and the empirically measured cross-rig performance degradation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scene Metadata<br/>Map/Weather/Traffic Density"] --> B["Initial CARLA Scene<br/>Record Entity Trajectories"]
    B --> C["Trajectory Pruning<br/>Reduce Near-Static Oversampling"]
    C --> D["Deterministic Replay<br/>14 Rigs × Same 115 Scenes"]
    D --> E["PCCR Multi-Rig Dataset<br/>6,900 Samples, 724,500 Images<br/>9-Class 3D Annotations"]
    E --> F["Train Models on Source Rig<br/>BEVDet/BEVFusion<br/>Fast-BEV/PETR"]
    F --> G["Evaluate on Target Rig<br/>Relative mAP Drop Matrix"]
    D --> H["Rig Calibration Metadata<br/>Camera Pos/Ori/FOV"]
    H --> I["RigV: Intra-Rig<br/>Geometric Diversity"]
    H --> J["RigCD: Inter-Rig<br/>Geometric Discrepancy (with components)"]
    I --> K["Correlation Analysis<br/>Spearman Rank Correlation"]
    J --> K
    G --> K
    K --> L["Calibrated RigCD Model<br/>Predict Cross-Rig Transfer Difficulty"]

Key Designs

1. PCCR Benchmark: A Multi-Rig Simulation Dataset with Systematically Controlled Variables

Existing benchmarks either employ a fixed rig (e.g., nuScenes, Waymo) or conduct cross-dataset evaluations without controlling the scenes. The core design concept of this work is to render 14 different rigs across the same set of 115 scenes, thereby eliminating the confounding effect of scene content on cross-rig analysis. The 14 rigs are divided into two families: the controlled variant series (comprising R1 and its 5 variants: R1-r rotation, R1-t translation, R1-f FOV, R1-c6 fewer cameras, and R1-c10 more cameras) change only one geometric factor at a time for controlled ablation analysis; the diverse rig series (R2-R9) covers typical layouts of various vehicle types such as SUVs, sports cars, fire trucks, and EU HGVs, reflecting differences in the number of cameras (4-10), mixed FOVs (\(65^\circ\)-\(120^\circ\)), and mounting positions (bumper, fenders, windshield, mirrors, and roof) to evaluate generalization. The data generation employs a two-stage pipeline to avoid non-deterministic interference: the first stage records the complete motion trajectories of all entities in scenes with dynamic traffic, and the second stage replays these trajectories deterministically. This ensures that the observations of different rigs for the same batch of scenes are pixel-aligned in both space and time. Trajectory pruning downsamples frames based on bins of the ego-vehicle's average speed, reducing the overrepresentation of near-static behaviors. Annotations cover 9 categories (car, truck, bus, motorcycle, bicycle, adult, child, traffic sign, traffic light) within an 80-meter range around the ego-vehicle, labeling only the objects actually visible to each rig to avoid ill-posed labels.

2. RigV: Intra-Rig Geometric Diversity Descriptor

The goal of Rig Variance is to characterize the "internal richness" of a single camera layout with a single scalar—the more dispersed the spatial distribution, the larger the orientation differences, and the more diverse the FOVs among cameras, the higher the RigV. Each camera \(\mathbf{c}_i\) is represented by a position \(\mathbf{t}_i\in\mathbb{R}^3\), an orientation \(\mathbf{r}_i\in SO(3)\), and an FOV \(f_i\). RigV calculates the normalized differences between all camera pairs within a rig and averages them: \(\text{RigV} = \frac{1}{N(N-1)}\sum_{i\neq j}(\lambda_t\Delta t_{ij}+\lambda_r\Delta r_{ij}+\lambda_f\Delta f_{ij})\), where the translation, rotation, and FOV components are normalized using the maximum pairwise values within the rig (e.g., \(\Delta t_{ij}=\|\mathbf{t}_i-\mathbf{t}_j\|_2 / D_{\max}\)) to ensure all components fall in \([0,1]\) scale. Rigs with high RigV (such as R5's 4-camera \(120^\circ\) wide FOV layout, or R6's 9-camera fire truck layout) exhibit high internal diversity and broad coverage. In experiments, it is observed that models trained on high-RigV rigs exhibit smaller cross-rig generalization variance—suggesting that the internal diversity of the training rig promotes cross-rig robustness to some extent.

3. RigCD: Inter-Rig Geometric Discrepancy Descriptor

Rig Contrastive Distance quantifies the geometric discrepancy between two rigs to predict the performance degradation when transferring from a source rig to a target rig. The core challenge is that the two rigs may have different numbers of cameras (ranging from 4 to 10), and there is no natural camera-to-camera correspondence. RigCD addresses this with a two-step approach: first, it constructs a cost matrix for the camera sets of the two rigs (representing weighted translation \(\|\mathbf{t}_i^A-\mathbf{t}_j^B\|_2\), geodesic rotation distance \(d_R(\mathbf{r}_i^A,\mathbf{r}_j^B)\), and absolute FOV difference \(|f_i^A-f_j^B|\) for each cross-rig camera pair). Next, it solves the optimal bipartite matching using the Hungarian algorithm, taking the average discrepancy of the matched \(M\) camera pairs as the matching term \(\text{RigCD}_{\text{match}}\). Simultaneously, a normalized penalty for camera count differences is applied: \(\text{RigCD}_{\text{count}} = |N_A-N_B|/\max(N_A,N_B)\). The final RigCD is a weighted sum of both terms: \(\text{RigCD}(A,B) = \alpha\,\text{RigCD}_{\text{match}}(A,B) + (1-\alpha)\,\text{RigCD}_{\text{count}}(A,B)\). This descriptor can also be decomposed into component-wise versions—\(\text{RigCD}_t / \text{RigCD}_r / \text{RigCD}_f / \text{RigCD}_{\text{count}}\)—to independently examine the influence of each geometric dimension.

4. Cross-Rig Evaluation Protocol and RigCD Calibration Framework

The evaluation protocol defines a comprehensive \(9\times9\) cross-rig transfer matrix: a model is trained on the training set of the source rig \(R_i\) and evaluated on the test set of all target rigs \(R_j\), computing the relative performance drop \(\Delta\text{mAP}_{\text{rel}}(R_i\!\to\!R_j) = (\text{mAP}(R_i\!\to\!R_i)-\text{mAP}(R_i\!\to\!R_j)) / \text{mAP}(R_i\!\to\!R_i)\). The weight parameters \(\lambda_t\), \(\lambda_r\), \(\lambda_f\), \(\alpha\) of RigCD, along with a global scaling factor \(K\), are calibrated and optimized using L-BFGS-B on the controlled variant rigs (R1 and its 5 variants, 6 rigs in total). The optimization objective is to maximize the Spearman rank correlation coefficient between the predicted value \(K\cdot\text{RigCD}(R_i,R_j)\) and the empirically measured \(\Delta\text{mAP}_{\text{rel}}\). The calibrated model does not access any data from the diverse rigs (R2-R8) during calibration, and it is directly evaluated on them to assess its ranking prediction capabilities—this strict train/test separation ensures a fair evaluation of generalization. Finally, the Spearman \(\rho\) and the 95% confidence interval computed via 5,000 bootstrap runs are reported.

Loss & Training

Four baseline models (BEVDet, BEVFusion, Fast-BEV, and PETR) are implemented based on official MMDetection3D repositories, with only the dataloader modified to adapt to the PCCR format. Training uses the AdamW optimizer on two NVIDIA RTX 6000 GPUs with a batch size of 4 per GPU. The input resolution varies slightly across models: BEVDet at 384×704 (24 epochs), BEVFusion at 256×704 (20 epochs), Fast-BEV at 320×576 (40 epochs), and PETR at 320×800 (120 epochs). All models use camera input only, predict the 9 classes of PCCR targets, and disable the velocity prediction head.

Key Experimental Results

Main Results

The following table presents the in-domain mAP and the cross-rig transfer difficulty ranking performance (Spearman \(\rho\) between RigCD and empirically measured \(\Delta\text{mAP}\) on the diverse rigs R2-R8) for the four baseline models:

Model	In-Domain mAP (Average across all rigs)	RigCD \(\rho\) (Diverse Rigs)	95% CI
BEVDet	0.155 ± 0.053	0.718	[0.620, 0.791]
BEVFusion	0.163 ± 0.052	0.734	[0.649, 0.800]
Fast-BEV	0.090 ± 0.043	0.097	[-0.054, 0.235]
PETR	0.149 ± 0.049	0.804	[0.735, 0.851]

BEVFusion demonstrates the best in-domain performance (mAP 0.163), but PETR is the most reliable in terms of predicting the ranking of cross-rig transfer difficulty (\(\rho=0.804\)). The overall performance of Fast-BEV is significantly inferior to the other models (in-domain mAP of only 0.090), and its behavior under rig variation can hardly be explained by the geometric descriptor (\(\rho=0.097\), with the confidence interval crossing zero). Calibrated RigCD weights reveal the relative importance of each geometric factor: the translation coefficient \(\lambda_t=2.18\!-\!2.43\) (the highest), the rotation coefficient \(\lambda_r\) is moderate for BEVFusion (0.54) and PETR (0.61) but lower for BEVDet (0.35), and the FOV coefficient \(\lambda_f=0.11\!-\!0.29\) (the lowest). The cross-rig transfer matrix (Figure 3) shows that when the FOV goes out of the training distribution (R1-f), BEVDet, BEVFusion, and PETR nearly collapse. Among the diverse rigs, R6 (fire truck layout) and R9 (truck layout) pose the greatest challenges to Fast-BEV and PETR.

Ablation Study

Ablation results for individual components of RigCD reveal architecture-level differences among models:

Configuration	BEVDet \(\rho\)	BEVFusion \(\rho\)	PETR \(\rho\)
Full RigCD	0.718	0.734	0.804
w/o Translation (\(\lambda_t=0\))	0.743	0.719	0.220
w/o Rotation (\(\lambda_r=0\))	0.719	0.738	0.808
w/o FOV/Overlap (\(\lambda_f=0\))	0.330	0.331	0.787
w/o Camera Count (\(\alpha=1\))	0.716	0.733	0.800

For BEVDet and BEVFusion (both utilizing LSS-style view transformers), FOV/overlap consistency is a key factor—removing \(\lambda_f\) leads to a sharp drop of approximately 0.4 in \(\rho\). For PETR (a DETR-style query-based approach), translation is the most critical—excluding \(\lambda_t\) causes \(\rho\) to plunge from 0.804 to 0.220. The reason for this divergence is that LSS-based methods rely on accurate viewing frustum projection to the BEV grid (FOV changes directly violate the assumptions of the projection matrix), whereas DETR-style methods rely on mapping 3D position embeddings to image features (translation alters spatial correspondences). In multi-model joint calibration experiments, a shared set of RigCD weights achieves a ranking prediction capability of \(\rho > 0.59\) across all four models (improving Fast-BEV from 0.097 to 0.589), verifying that RigCD carries a certain degree of model independence. Ablation of multi-rig joint training shows that BEVFusion trained on non-overlapping data from 6 rigs generalizes better on average to 3 unseen rigs than the worst single-rig transfer, but falls short of the best single-rig transfer.

Key Findings

• The cross-rig domain gap is significant and purely geometrically explainable: Under completely identical scene content, merely altering rig geometry causes the mAP of BEVDet/BEVFusion/PETR to nearly collapse under extreme FOV changes; RigCD predicts this transfer difficulty with an accuracy of \(\rho=0.80\).
• Translation discrepancy is the dominant factor in cross-rig generalization: Consistent evidence from three reliable models shows that changes in camera mounting positions have the greatest impact on performance, followed by FOV/overlap modifications, while rotation changes have relatively minor impacts.
• Architectural rig robustness correlates strongly with the representation paradigm: LSS-based methods are extremely sensitive to FOV (due to depth frustum projection relying on calibration), query-based PETR is sensitive to translation (due to position embeddings tied to the world coordinate system), and Fast-BEV's rig behavior is barely predictable from geometric factors.
• Training on a high-RigV rig improves generalization variance: The higher the internal diversity of the training rig, the lower the variance of cross-rig transfer, though it may not yield the best absolute performance.
• RigCD can serve as an evaluation tool in the rig design stage: Without training any models, calibration parameters alone can provide a meaningful ranking of transfer difficulty between different rig configurations.

Highlights & Insights

• The scene-consistent 14-way cross-render design is the most core methodological contribution. By recording trajectories prior to deterministic replay, 14 rigs obtain pixel-aligned, identical-scene data. This is significantly more rigorous than prior cross-dataset or cross-scene cross-rig analyses, allowing performance degradation caused by the rig to be directly attributed without scene confounding.
• RigCD achieves a transfer difficulty ranking correlation of \(\rho=0.80\) using only calibration metadata, implying that perception models' behaviors under rig changes are largely geometrically predictable and model-independent. This has direct instructional value for rapid evaluation during rig selection and design: cross-rig compatibility of different configurations can be compared without running any models.
• The sensitivity divergence between LSS and query-based designs (FOV vs. translation) in the ablation study exposes fundamental differences between the two paradigms, offering architectural design insights for rig-aware models: a intuitive path is to explicitly inject rig calibration parameters into intermediate network representations, enabling the model to ignore geometric fluctuations and focus on semantics.
• The "two-stage" evaluation protocol—calibrating on controlled variants and validating on diverse rigs—is rigorously designed to avoid overfitting the metrics. Joint calibration experiments further suggest that even behaviors on "eccentric" architectures can represent improved correlation when calibrated with more diverse data points, bringing hope to universal modeling of rig discrepancy across architectures.

Limitations & Future Work

• PCCR is based on the CARLA simulator, inherently entailing a sim-to-real gap. Although pure geometric effects themselves are relatively insensitive to rendering fidelity, models' cross-rig behaviors under real-world sensor noise and photometric perturbations might deviate from simulation findings.
• Only four 3D detection architectures are evaluated, leaving tasks like segmentation, tracking, and occupancy prediction untouched, as well as VLM-based methods. Given the compact backbone (ResNet-50) and relatively short training schedules used for baseline experiments, the robustness rankings might differ with stronger backbones.
• RigCD only models geometric factors, neglecting non-geometric rig discrepancies such as lens distortion, rolling shutter, and sensor noise. Furthermore, RigCD is defined as a pairwise comparison and cannot be easily generalized to more complex transfer relationships in multi-rig joint training scenarios.
• The improvement of Fast-BEV from \(\rho=0.097\) to \(0.589\) in the joint calibration experiment indicates that some models' behaviors under rig variations require calibration with more data points to be captured by geometric descriptors, likely because the relationship between their rig degradation patterns and geometric factors is more complex or highly non-linear.

Related Work & Insights

• vs. Fixed-Rig Datasets (nuScenes, Waymo): These datasets laid the foundation for standardized 3D detection evaluation, but fixed rigs prevent the study of cross-rig generalization. PCCR expands upon them by introducing the rig dimension and adopts a nuScenes-compatible data format to lower user barriers.
• vs. Viewpoint Robustness Evaluation (Klinghoffer et al., Embacher et al.): The former only applies camera viewpoint perturbations to a single rig, while the latter only tests rig modifications across two vehicle models, providing limited coverage of the rig space. PCCR encompasses 14 rigs (including various vehicle models, mixed FOVs, numbers of cameras, and mounting positions), offering a significantly more complete rig design space.
• vs. Perception Entropy (Ma et al.): Perception entropy measures information gain in sensor configurations via uncertainty reduction but relies on model outputs and is computationally expensive. RigCD, which only requires calibration metadata, is highly suited for rapid estimation during the rig design phase.
• vs. Rig3R: Rig3R utilizes camera rig priors in 3D reconstruction but is not open-source, and its rig robustness for semantic perception tasks remains unknown. This work provides a reproducible evaluation baseline and a geometric understanding.

Rating

• Novelty: ⭐⭐⭐⭐ Cross-rig generalization is a core pain point in production deployment but has long been overlooked in academic research; this study fills the gap by introducing the first systematic benchmark and geometric descriptors.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 14 rigs × 4 architectures with a comprehensive transfer matrix. It includes validation on controlled variants, diversity-driven generalization, component ablation, joint calibration, and multi-rig training, making the design exceptionally rigorous.
• Writing Quality: ⭐⭐⭐⭐⭐ The problem motivation is clear, benchmarks and metrics are thoroughly defined, the experimental results are explained with depth, and the discussion is honest.
• Value: ⭐⭐⭐⭐ Offers direct guidelines for sensor selection and the design of rig-aware architectures, although the sim-to-real gap and limited task scope restrict its immediate production value.

Related Papers

• [CVPR 2026] RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding
• [ECCV 2026] CooperScene: Multi-Modal Cooperative Autonomy Benchmark with C-V2X Communication Characterization
• [ECCV 2026] RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception
• [CVPR 2026] CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction in Challenging Road Topography
• [ICML 2026] TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models