Skip to content

QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception

Conference: ECCV 2026
arXiv: 2509.03704
Code: https://github.com/ucla-mobility/QuantV2X
Area: Multi-Agent Cooperative Perception / Model Quantization
Keywords: V2X Cooperative Perception, Model Quantization, Multi-Agent Systems, Post-Training Quantization, Codebook Communication

TL;DR

QuantV2X proposes the first fully quantized multi-agent V2X cooperative perception system. By utilizing end-to-end quantized neural network models and quantized communication message representations, it maintains 99.8% of the perception accuracy of full-precision models under low-bit INT4/INT8 quantization. Simultaneously, it reduces end-to-end system latency by 3.2 times and surpasses the full-precision baseline in mAP30 by +9.5 on the V2X-Real dataset.

Background & Motivation

Background: V2X cooperative perception overcomes the blind spots and occlusion limitations of single-vehicle vision by enabling real-time sharing of sensor information among autonomous driving agents. The mainstream paradigm is intermediate fusion, where each agent encodes raw sensor data into BEV features to be shared, and the receiver then fuses them for detection. In recent years, research has primarily focused on improving fusion accuracy metrics.

Limitations of Prior Work: Almost all existing systems operate on the assumption of full-precision (FP32) computation. This leads to three systemic bottlenecks: (1) high computational overhead of full-precision inference, resulting in high local inference latency on vehicle-mounted GPUs; (2) massive data volume of FP32 BEV feature map communication (approx. 8.6 MB/frame for typical sizes), causing severe transmission delay; and (3) high GPU memory usage of full-precision models and feature maps, limiting the amount of cooperative historical information that can be cached. The combination of these three factors sharply degrades the performance of full-precision systems in real-world deployments, rendering them non-functional under strict latency constraints.

Key Challenge: There is a fundamental mismatch between the accuracy improvements of existing research and the efficiency requirements of real-world deployment—papers pursue higher AP metrics while ignoring deployment constraints such as inference latency, communication bandwidth, and GPU memory budgets. Real-world testing platforms like V2X-ReaLO have demonstrated that the actual online deployment performance of full-precision systems is significantly lower than ideal offline evaluation results.

Goal: To construct a full-stack quantized cooperative perception system from the model side to the communication side, maintaining or even exceeding the perception capability of full-precision systems under deployment constraints. This is further decomposed into three sub-problems: (1) How to maintain cooperative perception accuracy under low-bit quantization? (2) How to compress communication messages to reduce transmission latency? (3) Can the quantized system outperform full-precision baselines under realistic latency constraints?

Key Insight: The authors observe that full-precision representation dominates latency at both the computation and communication ends—model inference is slow due to the high cost of FP32 multiply-accumulate operations, and communication is slow due to the large size of FP32 feature maps. Quantizing both model weights/activations and transmitted messages to low-bit representations can reduce latency from both sources simultaneously.

Core Idea: Replace full-precision computation and full-precision message transmission with full-stack quantization (model-side PTQ + communication-side codebook quantized messages). Use heterogeneous alignment modules to compensate for the feature distribution shift introduced by quantization, enabling the low-bit system to outperform the full-precision baseline under realistic latency constraints.

Method

Overall Architecture

QuantV2X addresses the deployment efficiency challenges of full-precision V2X systems: slow local inference, large communication data volumes, and high memory footprints, all of which lead to latency explosions and sharp performance drops in real-world settings. The general approach is to quantize both the model and communication channels to low bit-widths (INT8/INT4), while introducing codebook message representations and alignment modules to ensure the quality of quantized features does not collapse. The system runs serially in three stages.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: RGB / LiDAR<br/>Multi-agent sensor data"] --> B["Stage 1: Full-Precision Pre-training<br/>Train FP32 intermediate fusion model"]
    B --> C["Stage 2: Codebook Learning<br/>Learn shared codebook, freeze encoder for initial training<br/>then jointly optimize the whole model"]
    C --> D["Stage 3: Post-Training Quantization DTQ<br/>Multi-agent sample calibration → Block-wise reconstruction<br/>→ Fusion layer alignment module"]
    D --> E["Deployment & Inference<br/>INT4/8 model inference<br/>+ Codebook index transmission"]

Stage 1 trains a standard FP32 intermediate fusion model (encoder + fusion module + detection head) to serve as the base for subsequent quantization. Stage 2 introduces a shared codebook to quantize BEV feature maps into codebook indices for communication compression, freezing the encoder first to train the codebook, followed by joint fine-tuning of the entire model. Stage 3 uses a small amount of calibration data (0.5% of the training set) to perform post-training quantization on the entire model, conducting block-wise reconstruction to minimize quantization error, and applying additional heterogeneous and spatial alignment losses on the fusion layers.

Key Designs

1. Codebook Quantized Message Representation: Replacing FP32 Feature Map Transmission with Codebook Indices

Communication bandwidth is the primary bottleneck in real-world V2X deployments—an FP32 BEV feature frame takes about 8.6 MB, which results in a transmission latency of over 2.5 seconds at a 27 Mbps transmission rate, failing real-time requirements. This design utilizes a learnable codebook \(\mathcal{D} = \{d_1, d_2, \ldots, d_{n_L}\} \in \mathbb{R}^{C \times n_L}\) shared among all agents to compress transmission. For each spatial location \((h,w)\), the BEV feature vector \(F_{[h,w]}\) is mapped to the nearest neighbor \(d_\ell\), and the sender transmits only the codebook index \(\text{index}_{[h,w]}\). The receiver reconstructs the feature \(\hat{F}_{[h,w]}\) using its local codebook.

The mathematical essence of bandwidth reduction is that the original FP32 transmission requires \(\log_2(H \times W \times C \times 32 / 8)\) bits, while codebook indices require only \(\log_2(H \times W \times \log_2(n_L) \times n_R / 8)\) bits. When \(n_L=128, n_R=1\), the message size is compressed from 8.6 MB to 0.03 MB, representing a compression ratio of ~287x, and the communication latency is reduced from 286.8 ms to 12.9 ms. Training is split into two phases: first, the encoder is frozen and only the codebook parameters are optimized to minimize reconstruction error \(\min_{\Theta_{\mathrm{cb}}} \sum_{(h,w)} \|F_{[h,w]} - \hat{F}_{[h,w]}\|_2^2\); second, the whole model is unfrozen and jointly optimized with the detection loss, teaching the encoder to generate features that are easier to reconstruct via the codebook. Compared to previous codebook methods like CodeFilling, this design co-designs codebook training with the subsequent PTQ phase, ensuring that quantized encoder features can still be reconstructed with high quality by the codebook.

2. Post-Training Quantization PTQ: Block-wise Reconstruction Calibration + Multi-Agent Sampling Strategy

Directly performing layer-wise linear quantization on an FP32 model causes severe accuracy loss, particularly in multi-agent heterogeneous scenarios where activation distributions across agents differ, making quantization ranges hard to calibrate. The core of this PTQ pipeline is block-wise reconstruction: the network is divided into functional blocks (e.g., backbone blocks, fusion blocks, detection head blocks). For each block, quantization parameters \(s, z\) are initialized using a Max-min strategy, and a set of candidates \(\{s_t\}_{t=1}^{T}\) is linearly discretized around the initial scale \(s_0\). The optimal scale \(s_{\text{opt}} = \arg\min_{s_t} \|X - \hat{X}(s_t)\|_F^2\) that minimizes the Frobenius reconstruction error is chosen. To further reduce rounding errors, AdaRound is introduced to adaptively select rounding directions for each weight element.

Key to this is how the calibration dataset is constructed: the authors propose a multi-agent sampling strategy that randomly samples various agent combinations and communication topologies to build the calibration set. This step exposes the quantization parameters to a sufficiently diverse range of collaboration patterns, preventing generalization collapse during actual deployment caused by overly homogeneous calibration data. With only 0.5% of the training data (5000 calibration steps), the accuracy of INT4/8 can be recovered to 98.8%-99.8% of FP32. Unlike LiDAR-PTQ (which only quantizes partially), QuantV2X quantizes the entire pipeline—including the encoder, fusion module, and detection head—with a calibration overhead that is only 40% of LiDAR-PTQ's (0.38 vs. 0.93 GPU hours).

3. Alignment Module: Heterogeneous Alignment Loss + Spatial Alignment Loss to Correct Quantization Shift

In multi-agent cooperative scenarios, directly quantizing the fusion module leads to two types of feature shifts: (1) systematic biases in activation distributions because different agents possess different sensor modalities (LiDAR vs. camera) and encoder architectures (PointPillar vs. SECOND); (2) localization noise and transmission asynchrony in real deployment introduce spatial misalignment, and quantization further amplifies this sensitivity.

The alignment module acts on the fusion layer during the PTQ calibration phase, applying two additional losses. The heterogeneous alignment loss uses KL divergence to align the fusion BEV feature distributions of the full-precision and quantized models: \(\mathcal{L}_{\text{hetero}} = D_{\text{KL}}(\mathbf{H}_i^{\text{fp}} \| \mathbf{H}_i^{\text{int}})\), driving the quantized fusion features to maintain a spatial activation pattern consistent with the full-precision model, which reduces false positive detections. The spatial alignment loss uses L2 distance to align the final bounding box outputs of the full-precision and quantized models: \(\mathcal{L}_{\text{spatial}} = \|\mathcal{B}_i^{\text{fp}} - \mathcal{B}_i^{\text{int}}\|_2^2\), directly constraining the spatial consistency of downstream detection results to boost localization accuracy in long-range and high IoU threshold scenarios.

As shown in Table 10, using AdaRound alone yields a long-range (50m+) AP70 of only 22.0. Adding the alignment module improves it to 27.9 (+27%), showing significantly better robustness to localization noise than naive quantization—under extreme pose errors, naive quantization AP30 drops from 75.1 to under 40, while QuantV2X maintains a performance level close to full precision.

A Complete Example

Let us walk through the inference process of a two-agent collaboration scenario (a vehicle with LiDAR-PointPillar + a roadside unit with LiDAR-SECOND). The input consists of one LiDAR point cloud frame from each side (~60,000 points/frame). The encoder produces an FP32 BEV feature map of size \(H \times W \times C = 200 \times 200 \times 256\), which would originally require transmitting \(200 \times 200 \times 256 \times 4 \approx 40.96\) MB (assuming no compression). After codebook quantization (\(n_L=128, n_R=1\)), the 256-dimensional vector at each spatial position is replaced by a 7-bit index, transmitting only \(200 \times 200 \times 7/8 \approx 35\) KB of index data plus the codebook (\(128 \times 256 \times 4 \approx 128\) KB, distributed once), totaling around 0.03 MB of transmission volume. The receiver uses the local codebook to reconstruct \(\hat{F}\) via lookup, passing it into the INT8 quantized fusion module, where the fusion features calibrated by the alignment module have nearly identical distributions to the FP32 version, and the detection head outputs the 3D bounding boxes. The end-to-end system latency is reduced from the full-precision system's 346.3 ms (local 29.2 + fusion 30.3 + communication 286.8) to 40.0 ms (local 14.2 + fusion 12.9 + communication 12.9), a reduction of 88%. The key is not just faster computing, but that communication latency has been transformed from a massive bottleneck into a negligible overhead.

Loss & Training

The system is trained in three stages, each with independent optimization targets:

  • Stage 1 (Full-Precision Pre-training): Standard detection loss \(\mathcal{L}_{\text{det}}\) trains the FP32 intermediate fusion model, for 40 epochs; the best checkpoint is selected.
  • Stage 2 (Codebook Learning):
    • Sub-stage 1 (Freeze encoder, 20 epochs): \(\min_{\Theta_{\mathrm{cb}}} \sum_{(h,w)} \|F - \hat{F}\|_2^2\), training the codebook with pure reconstruction loss.
    • Sub-stage 2 (Joint fine-tuning, 10 epochs): \(\min_{\theta, \mathcal{D}} \mathcal{L}_{\text{det}}(\hat{B}, B^{\text{gt}}) + \lambda_{\text{rec}} \sum_{(h,w)} \|F - \hat{F}\|_2^2\), joint optimization of detection loss + reconstruction loss to adapt the encoder to the codebook.
  • Stage 3 (PTQ Calibration, 5000 steps): For each block, optimize quantization parameters via Frobenius reconstruction error in Eq. (8), while applying \(\mathcal{L}_{\text{hetero}}\) and \(\mathcal{L}_{\text{spatial}}\) at the fusion layer. The calibration set is constructed through random multi-agent sampling and represents only 0.5% of the training dataset.

Key Experimental Results

Main Results

System-level performance comparison (V2X-Real dataset). When incorporating system latency into the evaluation, QuantV2X significantly outperforms all full-precision and communication compression baselines:

System Transmitted Feature / Volume mAP30/50 Gap with Upper-bound \(\Delta\)
Upper-bound (Ideal, zero-latency) - 53.8/43.5 -
Full-Precision (No compression) BEV Feature / 8.6 MB 43.1/34.8 -10.7/-8.7
Full-Precision (16x compressed) BEV Feature / 0.54 MB 48.8/38.0 -5.0/-5.5
Where2Comm BEV Feature / 0.30 MB 49.7/39.0 -4.1/-4.5
CodeFilling Codebook / 0.03 MB 51.4/40.8 -2.4/-2.7
QuantV2X (Ours) Codebook / 0.03 MB 52.6/42.2 -1.2/-1.3

System latency decomposition: QuantV2X's total latency is 40.0 ms (computation 27.1 ms + communication 12.9 ms), while the full-precision system's is 346.3 ms (computation 59.5 ms + communication 286.8 ms), achieving an 8.7x latency reduction. Model quantization alone reduces computation latency from 59.5 ms to 27.1 ms (-55%), and owing to the halved activation bit-precision, communication latency also drops from 286.8 ms to 87.0 ms. Codebook communication further compresses communication latency to 12.9 ms.

Cross-fusion method generalization (DAIR-V2X, INT4/8, AP30/50):

Bits (W/A) Pyramid Fusion F-Cooper AttFuse V2X-ViT Who2com Where2comm
32/32 75.1/68.2 64.5/56.0 68.8/63.1 57.4/49.5 63.2/57.3 62.1/53.7
4/8 74.2/66.7 57.4/49.5 66.6/60.8 29.9/8.8 57.2/52.8 60.4/51.5

CNN-based fusion methods (Pyramid Fusion, Who2com) are the most robust to quantization; V2X-ViT collapses because LayerNorm and window attention are highly sensitive to quantization; F-Cooper's max-pooling is sensitive to outliers, yielding larger losses post-quantization.

Ablation Study

Component ablation (DAIR-V2X, INT4/8):

Configuration \(L_P\)+\(C_R\) AP30/50 \(L_P\)+\(L_S\) AP30/50 Accuracy Recovery Rate
Full-Precision 75.1/68.2 80.3/76.1 100%/100%
Max-min (Naive Quantization) 73.2/61.5 76.5/60.1 97.4%/90.2%
+ AdaRound 72.8/65.1 80.1/74.2 96.9%/95.4%
+ \(\mathcal{L}_{\text{hetero}}\) 74.0/66.4 80.8/75.3 98.5%/97.3%
+ \(\mathcal{L}_{\text{spatial}}\) 74.2/66.7 80.2/75.5 98.8%/99.8%

The alignment module is crucial for recovering AP50—under naive quantization, AP50 drops from 68.2 to 61.5 (-6.7), whereas adding the two alignment losses recovers it to 66.7 (only -1.5). This shows that quantization primarily damages high-IoU precise localization capability, which the alignment loss directly addresses.

Codebook hyperparameter ablation (V2X-Real): \(n_L=128, n_R=1\) yields the best system accuracy of 52.6 mAP30; increasing \(n_R\) to 2 causes system accuracy to drop (52.4) due to the increased message volume, which confirms the dominant impact of communication latency on final performance.

Pose noise robustness: Under 0-0.6m position noise and 0-1.0 degree orientation noise, QuantV2X's AP30 curve almost overlaps with FP32; naive quantization's AP30 drops sharply from 75 to around 40.

Key Findings

  • Communication latency is the primary system bottleneck: Compressing communication alone (CodeFilling, without model quantization) improves by +8.3 mAP30 over uncompressed full precision; model quantization alone improves by +5.6; combining both (QuantV2X) yields a +9.5 boost, indicating a synergistic effect between both optimizations. Fig. 6 clearly shows this compounding relationship in latency decomposition.
  • The alignment module is vital for long-range detection: 50m+ AP70 is improved from 22.0 (with AdaRound) to 27.9 (+27%), showing that quantization errors are geometrically amplified in long-range scenarios, which the alignment loss mitigates by directly constraining downstream bounding boxes.
  • INT4/8 is the practical sweet spot: Under 2-bit weight quantization, AP30 collapses from 75.1 to 40.8, and inference engines like TensorRT do not support 2-bit quantization, lacking deployment feasibility.
  • Scalability under GPU memory constraints: Under constrained GPU memory budgets (simulating vehicle-mounted Orin/Jetson), QuantV2X can load backbones 2-3x larger than those of full precision, achieving higher perception accuracy on the same hardware—a deployment paradigm advantage not reflected in offline accuracy metrics.
  • Real ROS testbed validation: In V2X-ReaLO online evaluation (including V2V, I2I, and V2I modes), QuantV2X's mAP is close to or even exceeds the ideal offline baseline of full-precision systems while significantly of outperforming online full-precision deployments (V2V mAP 50.1 vs. 46.1).

Highlights & Insights

  • Full-stack quantization thinking: Instead of isolating model quantization or communication compression, QuantV2X recognizes that reducing model precision also reduces communication volume (going from FP32 to INT8 activations naturally cuts communication in half), and that codebook indices are tighter than compressed FP32 features—leading to a multiplying effect. This "model-side quantization reduces communication; communication-side codebook squeezes it to the limit" design philosophy is highly translatable to other distributed perception systems.
  • Clear and complementary division of labor in the two losses of the alignment module: \(\mathcal{L}_{\text{hetero}}\) performs distribution alignment in the BEV feature space (resolving false positives and feature blurring), while \(\mathcal{L}_{\text{spatial}}\) conducts regression alignment on final detections (resolving localization accuracy and long-range degradation)—as clearly mapped out in the ablation study.
  • The counter-intuitive insight that "full precision can be inferior to model quantization under system latency constraints": This is the core insight of the paper—in real-world V2X scenarios, timeliness is more important than precision. Although quantization trades away a sliver of model accuracy, the low latency it unlocks ensures that cooperative information arrives strictly within the time window, ultimately boosting system-level performance. This establishes a new paradigm for evaluating cooperative perception systems.
  • Multi-agent sampling for calibration data construction: Randomly sampling agent combinations to build calibration sets ensures the quantization parameters learn enough cooperative variance—a simple but critical engineering detail, without which calibrated parameters drift significantly when encountering unseen agent combinations.

Limitations & Future Work

  • Evaluated only on INT8/INT4: 2-bit quantization experiments show performance collapse (AP30 dropping from 75.1 to 40.8). The paper did not explore mixed-precision strategies (e.g., maintaining high precision for sensitive layers and aggressive quantization for non-sensitive ones), which could be a path for further compression.
  • Fixed, globally shared codebook: The assumption of a shared codebook of size \(n_L=128\) assumes that BEV feature distributions across all situations are similar. In cross-domain (urban vs. highway vs. rural) scenarios, adaptive or hierarchical codebooks might be needed.
  • Evaluated only on detection tasks: The final goal of cooperative perception is planning and control; the impact of quantization on downstream planning modules remains unassessed. Retaining detection accuracy does not inherently guarantee planning safety.
  • Limited coverage of heterogeneous agent configurations: The experiments mainly cover LiDAR+LiDAR or LiDAR+Camera modal combinations, leaving out camera-only or radar combinations.
  • Latency model assumes a static 27 Mbps transmission rate: In real-world V2X communication (C-V2X/DSRC), bandwidth changes dynamically, and transmission rates can fall far below nominal values during channel congestion. Although quantized messages should offer even larger advantages under extreme low-bandwidth conditions, a formal sensitivity analysis is lacking in this paper.
  • vs. LiDAR-PTQ: LiDAR-PTQ only quantizes partial networks and targets single-vehicle LiDAR perception, whereas QuantV2X enables end-to-end full-pipeline quantization for multi-agent fusion scenarios and resolves multi-agent heterogeneity via its alignment module. QuantV2X is also more calibration-efficient (0.38 vs. 0.93 GPU hours).
  • vs. CodeFilling: CodeFilling also uses a codebook to compress communication but operates strictly on FP32 models, bypassing model-side quantization. QuantV2X demonstrates that when models are also quantized, the codebook must be jointly trained to maintain reconstruction quality—which explains why CodeFilling's system-level performance (51.4 mAP30) is lower than QuantV2X's (52.6).
  • vs. Where2comm: Where2comm compresses communication by selecting key regions to transmit based on spatial confidence maps—a sparsification strategy focused on "what to transmit." QuantV2X's codebook belongs to a quantization strategy targeting "how to transmit." These two can be combined (first sparsification, then codebook quantization), representing a promising direction for future work.
  • vs. LLM quantization methods (GPTQ/AWQ/SmoothQuant): These methods are tailored for autoregressive Transformers and discrete tokens. Applying them directly to 3D sparse convolutions / BEV encoders / multimodal fusion in V2X causes severe accuracy loss, as V2X requires keeping spatial geometric consistency rather than semantic probability distributions.

Rating

  • Novelty: ⭐⭐⭐⭐ It is the first to introduce full-stack quantization to V2X cooperative perception. The combination of codebook communication + end-to-end PTQ + alignment module is novel, though individual techniques (PTQ, VQ, AdaRound) are not originally invented by the authors.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively evaluated on two real datasets (DAIR-V2X, V2X-Real) + one simulation dataset (OPV2V) + a real ROS testbed (V2X-ReaLO), covering both model-level and system-level metrics. It includes ablation studies, robustness analyses, latency decompositions, scalability experiments, and power measurements.
  • Writing Quality: ⭐⭐⭐⭐⭐ Problem statements are clear, motivation figure (Fig.1) is intuitive, the method matches the three-stage description organically, and the experimental design specifically answers three research questions (RQ1/RQ2/RQ3). The QA-style discussion in the appendix comparing V2X with LLM quantization is very helpful.
  • Value: ⭐⭐⭐⭐⭐ Directly addresses the gap between research and deployment in V2X cooperative perception, offering a complete engineering solution and open-source code. It sets a new "system efficiency prioritized over pure accuracy" evaluation paradigm for the field, carrying high practical value.