A Prediction-as-Perception Framework for 3D Object Detection

Conference: CVPR 2026 arXiv: 2603.12599 Code: To be confirmed Area: 3D Object Detection / Autonomous Driving Keywords: 3D detection, prediction-perception closed loop, query propagation, temporal fusion, autonomous driving

TL;DR

Inspired by the human cognitive pattern of "anticipating target locations before focusing attention," this work converts trajectory predictions from the previous frame into detection queries for the current frame, forming an iterative prediction-perception closed loop. Applied to UniAD, the framework achieves simultaneous improvements of +10% in tracking accuracy and +15% in inference speed.

Background & Motivation

Existing attention-based 3D detectors (e.g., DETR3D, StreamPETR, Sparse4D) randomly initialize a set of queries at each frame and update them via cross-attention to perform detection. This paradigm suffers from two issues: (1) random queries lack positional priors, leading to low detection efficiency; (2) temporal information across frames is only implicitly propagated through attention, without explicit exploitation of predicted future object positions. In contrast, when humans perceive fast-moving objects (e.g., birds, insects), they first anticipate where the target will appear next and then direct attention accordingly — this "prediction-driven perception" mechanism is absent in existing detection frameworks.

Core Problem

Can 3D detectors, like the human brain, leverage predictions from historical frames to guide perception in the current frame? Specifically, can the output of trajectory prediction (future positions) be fed back into the detection module as initialization queries, enabling the detector to "know where to look"? The value of this question lies in the potential to reduce redundant query computation while improving recall for moving objects, if predictions can effectively guide perception.

Method

Overall Architecture

The PAP framework consists of a Perception module and a Prediction module, with queries serving as the communication medium to form a closed loop. The workflow proceeds as follows: current-frame images + previous-frame prediction queries → Perception module outputs detection results and queries → Prediction module forecasts future positions from detection queries → predicted positions are encoded via embedding into queries stored in a query bank → retrieved by the Perception module in the next frame. For the first frame, all queries are randomly initialized.

Key Designs

1. Prediction Query Substitution: A subset of the randomly initialized queries in the Perception module is replaced by position queries output by the previous frame's Prediction module. These prediction queries carry object motion trend information and are closer to the true object positions than random queries, enabling faster attention convergence. Formally: \(q_i^T \in (q_{random}^T \cup q_{predict}^{T-1})\), where a network \(\phi_{ref}\) maps queries to box center hypotheses.
2. Prediction Result Embedding: The Prediction module outputs future position coordinates \(c_{predict}^T = \text{PRED}(\text{PECP}(c_i^T))\), which are transformed via an embedding layer \(\phi_{embd}\) into vectors matching the Perception module's query dimensionality: \(q_{predict}^T = \phi_{embd}(c_{predict}^T)\). These are stored in the query bank indexed by timestamp for use in subsequent frames.
3. Plug-and-Play Design: The Perception module can directly adopt any existing query-based detector (DETR3D, StreamPETR, etc.), and the Prediction module can adopt any existing trajectory prediction method; the two communicate solely through queries. The loss functions remain identical to the original models, requiring no additional loss design.

Loss & Training

PAP introduces no new loss terms; the Perception and Prediction modules each retain their original loss functions. Learning of prediction queries is accomplished naturally through joint optimization of both modules. Experiments are conducted on UniAD, where trajectory prediction queries from MotionFormer are embedded and merged with TrackFormer's Track Queries as input. Training is performed on 4× A100 GPUs, with all hyperparameters kept identical to the original UniAD for fair comparison.

Key Experimental Results

Dataset	Metric	UniAD+PAP	UniAD	Gain
nuScenes val	AMOTA↑	0.395	0.359	+10%
nuScenes val	AMOTP↓	1.22	1.32	−0.10
nuScenes val	Recall↑	0.493	0.467	+0.026
nuScenes val	IDS↓	826	906	−80
nuScenes val	Training Time	78h	91h	−14%
nuScenes val	FPS↑	16	14	+15%

Per-category results (UniAD+PAP): Car achieves the best performance (AMOTA 0.613), followed by Bus (0.465), with Trailer being the most challenging (0.330). ID Switches for Car and Pedestrian are 405 and 342, respectively, accounting for 90% of total IDS.

Ablation Study

• No formal ablation study is provided; the authors acknowledge in the limitations section that ablation experiments were not conducted due to time constraints.
• The sole empirical validation compares overall performance on UniAD with and without the PAP framework.
• Ablations on query substitution ratio, prediction horizon selection, and different perception/prediction backbone combinations are all absent.

Highlights & Insights

• The biologically inspired "prediction-as-perception" concept is intuitive and elegant, naturally feeding trajectory prediction outputs back into the detection module.
• Replacing random queries with prediction queries not only improves accuracy but also reduces training time (91h → 78h) and improves inference speed (14 → 16 FPS), demonstrating that position-informed queries are genuinely more efficient than random initialization.
• The plug-and-play design preserves the original model losses and is theoretically compatible with any query-based detector/predictor combination.

Limitations & Future Work

• Validation is limited to a single base model (UniAD), whose perception and prediction modules are not individually state-of-the-art, limiting the persuasiveness of the results.
• Absence of ablation studies: How does the query substitution ratio affect performance? Is full vs. partial substitution preferable? Which prediction horizon yields the optimal query?
• Prediction errors may accumulate: inaccurate predictions from the previous frame could generate misleading queries that degrade current-frame detection.
• The effect on static objects (e.g., parked vehicles, traffic cones) is unclear — such objects do not require anticipatory localization.
• Only tracking metrics (AMOTA/AMOTP) are reported; standard 3D detection metrics (NDS, mAP) are not evaluated.

Related Work & Insights

• UniAD [CVPR 2023]: The experimental backbone for PAP. UniAD itself features a perception → prediction → planning pipeline with unidirectional module communication; PAP adds a reverse prediction → perception link to form a closed loop, improving AMOTA from 0.359 to 0.395.
• StreamPETR [ICCV 2023]: Also leverages temporal information to enhance detection, but does so via temporal attention to establish cross-frame query associations; PAP differs by explicitly using predicted future positions as query initialization rather than relying on implicit attention propagation.
• HOP [ICCV 2023]: Employs historical object predictions to enhance temporal training for multi-view 3D detection, sharing conceptual similarity with PAP; however, HOP introduces auxiliary losses during training, whereas PAP maintains the prediction → perception closed loop at inference time as well.

The "prediction-guided perception" paradigm in PAP can potentially be extended to other tasks, such as using semantic segmentation predictions to guide the query distribution for next-frame detection. There is also a natural connection to world models: replacing the Prediction module with a world model's state predictor could enable richer perceptual guidance. The approach may further benefit small object detection, where anticipating object locations and allocating dedicated queries could improve recall for easily missed targets.

Rating

• Novelty: ⭐⭐⭐ The biologically inspired "prediction-driven perception" concept is compelling, though the technical realization is relatively straightforward (query substitution + embedding).
• Experimental Thoroughness: ⭐⭐ Only a single base model and dataset are evaluated with no ablation studies, making the experimental design insufficient.
• Writing Quality: ⭐⭐⭐ The framework description is clear, but the paper is overall brief and lacks technical detail.
• Value: ⭐⭐⭐ The proposed closed-loop framework has broad applicability, but the current depth of validation is insufficient to be fully convincing.

Related Papers

• [CVPR 2026] R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection
• [CVPR 2026] On the Feasibility and Opportunity of Autoregressive 3D Object Detection
• [CVPR 2026] CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection
• [CVPR 2026] Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction
• [CVPR 2026] ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction