BeyondSight: Object Permanence for End-to-End Autonomous Driving¶
Conference: ECCV2026
Official Paper: ECCV 5981
Paper: PDF
Project: BeyondSight
Area: Autonomous Driving
Keywords: Object permanence, partial observability, temporal query propagation, sparse scene representation, occlusion-aware planning
TL;DR¶
BeyondSight separates historical actor propagation from current-image updates in SparseDrive and trains with annotations spanning unobservable intervals, keeping fully occluded actors available to prediction and planning while raising unobservable-actor mAP from 0 to 0.249 and reducing standard nuScenes average open-loop planning L2 from 0.61 to 0.54.
Background & Motivation¶
A driving model can aggregate historical features without understanding that an actor still exists when it disappears from view. Consider a pedestrian approaching a crossing and then becoming hidden behind another vehicle: the detector may suppress or remove the pedestrian query when visual evidence disappears, the predictor stops supplying its future trajectory, and the planner receives a scene missing a relevant participant. Sparse-query systems such as SparseDrive propagate instances across frames, but historical queries can still be eroded by prolonged occlusion when their updates remain centered on current image evidence.
Supervision and evaluation reinforce this behavior. Standard protocols commonly filter actors without sensor-point support, so a model retaining a genuinely present hidden actor can be penalized as producing a false positive. The paper approximates observability by whether an actor has any associated LiDAR or radar returns. This is an annotation-side proxy for sensor support, not an exact test of camera visibility, and does not mean the camera-based model receives those point clouds at inference. The authors estimate that roughly 30% of actors are fully unobservable at a timestamp; adding short-term temporal fusion alone cannot recover objects systematically excluded from supervision.
The task concerns actors observed previously and subsequently becoming temporarily unobservable, not discovering arbitrary actors that have never been seen. Core idea: support actor existence through persistent historical hypotheses, let current observations contribute fresh evidence separately, and make persistence learnable and measurable with annotations and evaluation that span observability gaps.
Method¶
Overall Architecture¶
BeyondSight retains SparseDrive's multi-view image backbone, sparse scene representation, motion prediction, and planning modules. Its central change is a three-stage actor-query update: a Temporal Prior Decoder produces image-independent hypotheses from historical detection and motion information; an Observation Decoder combines the prior, fresh queries, and current image features; and a Posterior Fusion Decoder reconciles both branches before supplying the final actor set to prediction and planning.
The accompanying nuScenes-Permanence extension supplies offline-completed actor states during unobservable intervals. These supervise propagated and fused hypotheses during training and support observability-conditioned evaluation. The terms prior and posterior describe filtering-inspired latent query representations, not calibrated probability distributions; classification confidence should not be read as a Bayesian probability of existence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
DATA["Permanence Annotations<br/>and Evaluation"] -.->|Training supervision| PRIOR
HISTORY["Historical detection<br/>and motion queries"] --> PRIOR["Temporal Prior Decoder"]
PRIOR --> OBS["Observation Decoder"]
IMAGE["Current image features<br/>and fresh queries"] --> OBS
PRIOR --> FUSION["Posterior Fusion Decoder"]
OBS --> FUSION
DATA -.->|Training supervision| FUSION
FUSION --> OUTPUT["Persistent actor set<br/>Motion prediction and planning"]
Key Designs¶
1. Permanence Annotations and Evaluation: make hidden actors legitimate supervision targets
nuScenes-Permanence preserves observed trajectory segments and retains existing boxes with no LiDAR or radar support. Missing states are reconstructed through interpolation between observed states or short-horizon extrapolation when an actor leaves the sensor field of view. Completion runs strictly offline: inference cannot look up completed trajectories or access future observations. Full sequences can therefore support reference construction during dataset preparation, while the deployed model must maintain actors from its own historical state. The extension targets approximately continuous motion, not reliable reconstruction of abrupt hidden stops or turns.
Evaluation must change too, or maintaining a hidden actor remains a punishable prediction. Ground truth is partitioned into observable actors, unobservable actors, and their union, yielding mAP_obs, mAP_unobs, and mAP_all. For a subset metric, predictions are first matched against the target set; remaining predictions matching the other, ignored set are removed before accumulation. This symmetric ignore rule avoids cross-penalization. Hidden-actor matching additionally uses a tolerance that varies with unobservability duration and returns to the standard threshold as that duration approaches zero. It is calibrated through a holdout reconstruction study that masks originally observed trajectory intervals and compares their reconstructions with ground truth. The main text does not specify the tolerance function, so no mathematical form is invented here.
2. Temporal Prior Decoder: update historical actors without current-image suppression
Historical detection queries and motion-predicted queries are fused through a lightweight MLP, then refined by self-attention operating solely on queries. Detection history carries information about previously observed actors, while motion queries condition the propagation on predicted movement rather than merely caching an old feature vector. Crucially, this stage never accesses current image features: an absence of visual evidence does not directly prevent an actor hypothesis from being produced.
This addresses a weakness of ordinary query propagation. Passing an old query forward is insufficient if all retained information must subsequently pass through a visual update that treats the actor as absent. The independent prior preserves an alternative candidate for fusion. It does not guarantee correct states indefinitely, however: motion-conditioned hypotheses can drift during long gaps. Object permanence is not a license to retain every historical actor forever without a termination mechanism.
3. Observation Decoder: incorporate visible evidence while admitting new actors
The temporal prior is combined with newly initialized object queries and passed through the standard SparseDrive detection decoder. Current image features support observation-conditioned hypotheses; fresh queries admit actors not already represented in history, while propagated queries can absorb corrective evidence when an actor reappears. This branch is supervised only on observable ground-truth actors, so preserving a hidden actor does not require inventing visual features in an image region occupied by its occluder.
The division of responsibilities matters more than simply lengthening memory. The observation branch asks what current sensing supports, while the prior branch carries hypotheses justified by earlier evidence. The branches need not agree during occlusion; resolving that disagreement is the purpose of fusion. Observable labels come from annotation-side sensor support during training, not from privileged access to hidden ground truth during inference.
4. Posterior Fusion Decoder: carry persistent actors into downstream decisions
For each actor, fusion retains the proposal with higher classification confidence and applies a final query-refinement stage. Both branches can supply the selected hypothesis: current observations can correct visible actors, whereas the propagated prior can preserve an actor when new evidence is unavailable. This is not an analytical product of probability distributions, and the word posterior does not justify reconstructing a Kalman-filter-style equation absent from the described mechanism.
Detection supervision for the temporal and fusion branches covers both observable and unobservable actors, and motion supervision extends to the full actor set as well. The posterior decoder also predicts an observability state trained with binary cross-entropy. Actor existence and current sensor support thereby become distinct output semantics rather than two meanings compressed into one detection score. The planner itself is unchanged; its improved context consists of a more complete actor set and the trajectories predicted for those actors.
A Worked Example¶
Consider the occlusion cases discussed in Figures 1 and 3: a pedestrian has been observed and then becomes fully hidden by another vehicle. Historical detection and motion queries pass through the Temporal Prior Decoder to retain a hypothesis about the pedestrian's continuing motion. The current-image branch can update the occluding vehicle without necessarily supplying useful evidence for the pedestrian. Fusion can still select the prior-supported pedestrian hypothesis, allowing prediction to maintain its future trajectory and planning to leave additional clearance.
When the pedestrian reappears, the observation branch can correct the state. If the pedestrian abruptly changes intent while hidden, the prior may instead become inaccurate, exposing the method's boundary. This example explains the module interaction without assigning unreported speeds, hidden trajectories, or query-confidence values.
Loss & Training¶
Training follows SparseDrive's two-stage schedule: first learn sparse perception, then jointly optimize perception, motion prediction, and planning. The objective retains observation-branch detection, map, planning, and depth losses while incorporating modified detection and motion supervision over all actors. The modified detection objective also includes binary cross-entropy for observability with weight 1.0. The main text does not fully specify every underlying loss implementation, and the cached PDF extraction corrupts several displayed equations; the supervision responsibilities are described here without reconstructing those equations.
Optimization uses AdamW with weight decay 0.001, cosine learning-rate scheduling, and 500 warm-up iterations. Stage 1 and Stage 2 learning rates are 0.0004 and 0.0003; image-backbone learning-rate multipliers are 0.5 and 0.1, respectively. The additional decoders use self-attention, two-layer MLPs, and detection heads, adding less than 5% to the parameter count. That figure is not evidence of less than 5% inference-latency overhead.
Key Experimental Results¶
Main Results¶
Experiments use nuScenes' official 700/150 training/validation scene split, six cameras, keyframes at 2 Hz, and ten actor classes. The following selection comes from Table 1. L2 is measured in meters and collision rate, CR, in percent; all are open-loop metrics and lower is better.
| Method | L2 1 s | L2 2 s | L2 3 s | Average L2 | Average CR (%) |
|---|---|---|---|---|---|
| UniAD | 0.48 | 0.96 | 1.65 | 1.03 | 0.31 |
| VAD-Base | 0.41 | 0.70 | 1.05 | 0.72 | 0.22 |
| SparseDrive | 0.29 | 0.58 | 0.96 | 0.61 | 0.08 |
| BridgeAD | 0.28 | 0.55 | 0.92 | 0.58 | 0.08 |
| MomAD | 0.31 | 0.57 | 0.91 | 0.60 | 0.09 |
| BeyondSight | 0.26 | 0.51 | 0.85 | 0.54 | 0.07 |
Object permanence is more directly tested by the conditional metrics in Table 3 than by aggregate planning scores. Higher mAP and EPA are better; minADE is the minimum average displacement error across predicted trajectory candidates, so lower is better. EPA follows the existing end-to-end prediction accuracy protocol and is not a new metric introduced by this paper.
| Evaluation Set | Method | mAP | minADE | EPA |
|---|---|---|---|---|
| All actors | SparseDrive | 0.389 | 0.642 | 0.457 |
| All actors | BeyondSight | 0.413 | 0.582 | 0.481 |
| Observable actors | SparseDrive | 0.415 | 0.610 | 0.492 |
| Observable actors | BeyondSight | 0.421 | 0.625 | 0.496 |
| Unobservable actors | SparseDrive | 0.000 | 0.615 | 0.000 |
| Unobservable actors | BeyondSight | 0.249 | 0.479 | 0.285 |
Under the standard nuScenes protocol in Table 2, the ResNet50 models improve detection mAP from 0.415 to 0.427, NDS from 0.526 to 0.536, and AMOTA from 0.372 to 0.401. Standard mAP of 0.427 and conditional mAP_obs of 0.421 belong to different evaluation protocols and should not be interchanged.
Ablation Study¶
The following results are taken from Table 4. Components are added cumulatively within each annotation group, not removed independently. Average CR follows the percentage convention used for planning results.
| Training Annotations | Cumulative Configuration | mAP_obs | mAP_unobs | mAP_all | Average L2 | Average CR (%) |
|---|---|---|---|---|---|---|
| Standard nuScenes | SparseDrive | 0.415 | 0.000 | 0.389 | 0.61 | 0.08 |
| Standard nuScenes | + Temporal prior | 0.417 | 0.000 | 0.390 | 0.60 | 0.08 |
| Standard nuScenes | + Posterior fusion | 0.413 | 0.000 | 0.388 | 0.61 | 0.08 |
| nuScenes-Permanence | SparseDrive | 0.403 | 0.021 | 0.388 | 0.61 | 0.08 |
| nuScenes-Permanence | + Temporal prior | 0.406 | 0.126 | 0.399 | 0.58 | 0.08 |
| nuScenes-Permanence | + Posterior fusion | 0.415 | 0.194 | 0.405 | 0.56 | 0.07 |
| nuScenes-Permanence | + Unobs. supervision | 0.418 | 0.213 | 0.410 | 0.55 | 0.07 |
| nuScenes-Permanence | Full BeyondSight | 0.421 | 0.249 | 0.413 | 0.54 | 0.07 |
Key Findings¶
- Architecture and supervision are jointly necessary: temporal and fusion modules still yield zero unobservable-actor mAP with standard annotations. Under permanence annotations, the temporal prior raises mAP_unobs from 0.021 to 0.126, the largest individual step in this cumulative table, although this setup cannot isolate every component interaction.
- Not every metric improves: conditional observable-actor minADE changes from 0.610 to 0.625. The results do not support a blanket claim that forecasting becomes more accurate for every actor group.
- A high-occlusion subset contains 600 validation samples with occluded dynamic actors whose future trajectories approach within 12 m of the ego plan over a 4 s horizon. SparseDrive and BeyondSight obtain average L2 of 0.64 and 0.56, and average CR of 0.163 and 0.08, respectively. This is a specifically selected subset, not a representative score for all traffic situations.
Highlights & Insights¶
- An image-independent state-update path. Retaining historical tokens is insufficient if every visual update can erase them. An independent propagation branch is a useful design pattern for embodied tasks with interrupted observations.
- Supervision and evaluation change together. Removing hidden actors from training and penalizing their predictions at evaluation leaves no incentive for persistence. The symmetric ignore rule makes conditional metrics better aligned with the intended capability.
- Planning gains come from the scene representation. Keeping the planner unchanged strengthens the interpretation that missing participants affect decisions. It does not, by itself, establish improved real-world driving safety.
Limitations & Future Work¶
- The authors acknowledge stale hypotheses and localization drift during long occlusions when actors deviate from the learned motion prior. Uncertainty-aware memory, termination criteria, and multiple motion hypotheses are natural next steps.
- Interpolation and short-horizon extrapolation provide hidden-state references rather than direct measurements. Abrupt stops, turns, and intent changes are not reliably covered; privileged simulation labels and multi-observer datasets could provide stronger validation.
- Matching tolerance varies with occlusion duration, while the exact completion and tolerance procedures are deferred to supplementary material absent from this cached main text. The reported 0.249 should not be described as hidden-actor performance under a single strict localization threshold.
- The paper reports an increase from 932k to 1.33M boxes and also labels it "+30%." Those endpoints do not match that percentage when the original count is the denominator. This note retains the endpoints and flags the ambiguity instead of repeating the growth percentage as settled.
- Table 4 does not sufficiently specify what separates "+ Unobs. supervision" from the final full-model row, or fully distinguish that step from the expanded annotation setting. The last increase from 0.213 to 0.249 cannot confidently be attributed to one particular loss.
- Evidence comes from open-loop evaluation on one dataset and qualitative examples. The main text supplies no closed-loop driving tests, confidence intervals, or measured latency; less than 5% extra parameters does not establish deployment cost.
Related Work & Insights¶
- SparseDrive is the direct baseline, using sparse instance queries for joint perception, prediction, and planning. BeyondSight primarily changes actor-query persistence and supervision coverage rather than replacing the entire driving stack.
- StreamPETR, ForeSight, and BridgeAD exemplify object-centric temporal propagation, memory sharing between detection and forecasting, and historical prediction supporting present tasks. BeyondSight specifically makes existence during fully unobservable intervals explicit in architecture, annotations, and evaluation.
- OPNet and PermaTrack already investigate object permanence in video understanding and tracking. The concept itself is therefore not new; this paper's contribution is to connect it to prediction and planning in an end-to-end driving system.
- Transferable lesson: separate whether an actor continues to exist from whether the sensor currently supports observing it. For robotics, that distinction still needs uncertainty and memory-expiration mechanisms so persistence does not become a source of long-lived false positives.
Rating¶
- Novelty: 4/5. The three-stage query mechanism is lightweight, but integrating persistence-aware supervision, modeling, and evaluation is valuable; object permanence has substantial precedent.
- Experimental Thoroughness: 3/5. Main tasks, conditional metrics, cumulative ablations, and an occlusion subset provide useful evidence, but closed-loop tests are missing, hidden references are reconstructed, and some ablation definitions remain unclear.
- Writing Quality: 4/5. The problem and branch responsibilities are clear, while the annotation-growth percentage and the final ablation step need clarification.
- Value: 4/5. The work targets a practical failure in which relevant road users disappear from the planning context, and highlights the limitations of visible-only evaluation. Ratings are the note author's assessment.