Skip to content

content_hash: 7928206726b3206a

Per‑Object IoU Forecasting for Deadline‑Aware Real‑Time Embedded Detection Control

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/3Eerfan/OLAP-ECCV2026
Area: Object Detection
Keywords: Streaming Perception, Real-Time Object Detection, IoU Decay Forecasting, Deadline-Aware Control, Embedded Edge Computing

TL;DR

This paper introduces an ultra-lightweight per-object IoU decay model (OLAP) and a closed-loop runtime detector selection controller (OLAP-C) for streaming detection, deriving closed-form per-object deadlines and optimizing priority-weighted Relative Average Precision to achieve a 56% improvement in deadline prediction accuracy and a 39% gain in streaming AP across Argoverse-HD and MOT17.

Background & Motivation

Streaming perception systems in autonomous driving, mobile robotics, and edge analytics are strictly bounded by real-time latency. While deep learning object detectors exhibit excellent offline detection accuracy, deploying them on resource-constrained embedded edge devices induces a non-negligible processing delay. Under continuous camera input, this delay produces substantial temporal mismatch—the physical elapsed time between when an input frame was sampled and when its corresponding detection outputs become available. As a result, bounding boxes lag behind fast-moving objects, causing catastrophic accuracy decay in real time. To mitigate this degradation, modern frameworks incorporate lightweight visual trackers (such as MOSSE or KCF) to propagate boxes across skipped frames, or deploy runtime controllers that dynamically select detector configurations or scheduling intervals.

However, existing closed-loop runtime controllers (e.g., ROMA, ARISE) rely exclusively on coarse frame-level feedback, such as frame-average predicted AP or frame-mean IoU decay. This coarse aggregation entirely masks substantial intra-frame heterogeneity across objects. In realistic dynamic scenes, even when the frame-average IoU remains around 0.73, high-speed objects crossing the field of view can suffer a complete drop to 0 IoU. In safety-critical applications, maintaining high spatial localization accuracy for an imminent crossing pedestrian is vastly more crucial than for distant static landmarks or parked vehicles. Managing control decisions solely with frame-level averages blinds the system to the urgent degradation of critical objects.

Furthermore, prior heavy data-driven runtime estimators demand gigabytes of memory and substantial inference overhead, rendering them impractical for resource-constrained edge hardware. Core Idea: Formulate a lightweight Generalized Linear Model predicting exponential IoU decay from 10-dimensional explicit object-level dynamic features, derive analytical closed-form deadline bounds for each object under user-specified safety thresholds, and schedule detector configurations via priority-weighted Relative Average Precision (RAP).

Method

Overall Architecture

The OLAP-C closed-loop control system executes at the completion instant of each streaming detection cycle. When detection cycle \(D_k\) finishes at frame \(F_i\), the controller measures its execution latency and conducts bipartite Hungarian matching with the streaming detections from the preceding cycle based on Generalized IoU (GIoU). For each matched object, a 10-dimensional feature vector encoding geometry, spatial dislocation, and normalized velocity is extracted. The OLAP model then predicts the per-object exponential IoU decay parameters, deriving closed-form deadline predictions under a user-specified IoU threshold. Finally, the controller optimizes detector selection by evaluating a dual-window Relative Average Precision metric that jointly accounts for the accuracy loss during the upcoming execution drop window and the anticipated gain across the post-inference evaluation horizon under scenario-driven object importance weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Streaming Frames and Prior Detection Box Inputs"] --> B["Cross-Cycle Object Association and 10D Dynamic Feature Extraction"]
    B --> C["Per-Object IoU Decay Forecasting and Analytical Deadline Derivation"]
    C --> D["Dual-Window Relative Average Precision Weighted Evaluation"]
    D --> E["Optimal Detector Selection and Closed-Loop Control Dispatch"]

Key Designs

1. Cross-Cycle Object Association and 10D Dynamic Feature Extraction: Capturing Object-Level Kinematics

To characterize object motion dynamics without running heavy neural estimators, the controller links the zero-mismatch detection output \(\tilde{B}_{i,j}\) (available at frame \(F_i\) from detector \(D_k\)) with the streaming bounding box \(B_{i-l_k, j}\) previously utilized at frame \(F_{i-l_k}\) using Hungarian matching with a GIoU cost matrix, establishing the paired correspondence \(M_{k-1,k}\). For each associated object, a compact 10-dimensional feature vector \(\mathbf{f}(B_{i,j}, B_{i-l_k,j}, l_{k-1}) = [f_1, \dots, f_{10}]^\top \in \mathbb{R}^{10}\) is constructed: - Geometry and spatial location: Aspect ratio \(f_1 = w/h\), geometric scale \(f_2 = \sqrt{wh}\), radial distance \(f_3 = \sqrt{(x_c-x_0)^2 + (y_c-y_0)^2}\) from image center \((x_0, y_0)\), and polar angle \(f_4 = \text{atan2}(y_c-y_0, x_c-x_0)\); - Spatial dislocation and motion: Non-overlapping spatial dislocation measured via Generalized IoU \(f_5 = \text{GIoU}(B_{i,j}, B_{i-l_k,j})\), normalized horizontal and vertical velocities \(f_6 = v_x/w, f_7 = v_y/h\), velocity magnitude \(f_8 = \sqrt{v_x^2 + v_y^2}\), velocity heading angle \(f_9 = \text{atan2}(v_y, v_x)\), and previous cycle execution latency \(f_{10} = l_{k-1}\). Feature sensitivity analysis confirms that velocity magnitude \(f_8\) and dislocation metric \(f_5\) carry the highest feature importance, directly mapping physical movement to localization decay.

2. Per-Object IoU Exponential Decay Modeling and Analytical Deadline Derivation: Zero-Search Temporal Bounds

Empirical observations reveal that the degradation of real-time IoU against temporal mismatch \(a_{i+s}\) follows a strict exponential decay. OLAP models this relationship for object \(j\) at future offset \(s\) via a two-parameter exponential formulation: $\(\widehat{\text{IoU}}_{i+s,j}^{\text{RT}} = \beta \cdot e^{-\gamma \cdot a_{i+s}}\)$ To maintain proper mathematical bounds (\(\beta \in (0, 1)\), \(\gamma > 0\)) without constrained optimization at runtime, a Generalized Linear Model (GLM) formulation maps features \(\mathbf{x}\) to \(\beta\) and \(\gamma\) using Sigmoid and Softplus link functions with trainable parameters \(\boldsymbol{\theta} = \langle \theta_0, \theta_1, b_0, b_1 \rangle\): $\(\begin{bmatrix} \beta \\ \gamma \end{bmatrix} = \begin{bmatrix} \text{Sigmoid}(\theta_0^\top \mathbf{x} + b_0) \\ \text{Softplus}(\theta_1^\top \mathbf{x} + b_1) \end{bmatrix}\)$ Given a safety-critical minimum acceptable IoU threshold \(\tau\) (e.g., 0.5 or 0.75), inverting the exponential formula yields the exact survival deadline \(C_q\) in closed form without iterative root-finding: $\(C_q = \frac{\ln(\beta / \tau)}{\gamma}\)$ This closed-form formulation dictates the maximum allowable frame delay before the current bounding box prediction becomes untrustworthy.

3. Dual-Window Relative Average Precision (RAP) Evaluation and Closed-Loop Control: Balancing Immediate Penalty and Future Utility

The controller manages a candidate detector set \(\mathcal{D} = \{d_1, \dots, d_M\}\), each characterized by an offline size-conditioned detectability profile \(P_m = [p^S_m, p^M_m, p^L_m]\) across small, medium, and large objects, alongside nominal inference latency \(\hat{l}_m\). At runtime, estimated inference latencies are calibrated proportionally to measured latency \(l_k\). To select the optimal detector for the upcoming cycle, OLAP-C evaluates candidate performance across two distinct temporal horizons: - Upcoming Drop Window: During the execution of candidate detector \(d_m\) spanning \(\hat{l}_m\) frames, no new detections arrive, and past boxes are reused over mismatch range \(a \in [l_k, l_k + \hat{l}_m]\). For each step \(a\), objects whose deadlines satisfy \(a < C_q\) count as true positives weighted by user-assigned importance \(\omega_q\). Averaging weighted precision across the drop duration yields \(\text{AP}_{\text{drop}}^m\); - Post-Inference Window: Following execution completion, evaluation continues over a horizon \(\bar{l}\) (the mean execution latency across all candidate detectors). The detector detectability scaling factor \(S_m = (P_m \oslash P_{m_k}) [n_S, n_M, n_L]^\top\) scales future precision estimates to form \(\text{AP}_{\text{future}}^m\). The controller sums these two components into a composite Relative Average Precision score: $\(\text{RAP}_m = \text{AP}_{\text{drop}}^m + \text{AP}_{\text{future}}^m\)$ The detector \(d_{m^*}\) maximizing \(\text{RAP}_m\) is selected for cycle \(D_{k+1}\), dynamically balancing the latency penalty of long inference against the detection quality of heavier backbones.

Loss & Training

The parameters \(\boldsymbol{\theta}\) of the OLAP model are trained offline on the first 60% of video sequences across datasets. To simulate diverse latency regimes, synthetic inference latencies \(l_{\text{sim}} \in \{1, 3, 5, \dots, 19\}\) are imposed on ground-truth boxes to create realistic tracking dislocation pairs across prediction offsets \(\mathcal{B} = \{0, 1, \dots, 29\}\). After standardizing features to zero mean and unit variance, \(\boldsymbol{\theta}\) is optimized by minimizing regularized Mean Squared Error (MSE): $\(\mathcal{L} = \frac{1}{N} \sum_{i,j} \epsilon_{i,j}^\top \epsilon_{i,j} + \lambda \|\boldsymbol{\theta}\|_2^2\)$ with regularization strength \(\lambda = 0.01\). Because the model relies strictly on universal geometric and kinematic dynamics, a single 6-hour CPU training run yields a robust predictor that generalizes across all candidate detection backbones and resolutions without retraining.

Key Experimental Results

Main Results

Streaming detection performance was evaluated on 71 video sequences from Argoverse-HD and MOT17 across three diverse embedded hardware platforms: an Intel Node (Core i7-10810U), an Apple M3 (MacBook Air), and an NVIDIA Jetson Orin Nano. Evaluations adopted standard COCO \(\text{sAP}@0.5:0.95\) metrics under native video frame rates with MOSSE tracking backend. Baselines include the state-of-the-art closed-loop controller ROMA and the Best Static detector configuration (YOLO11m_640).

Hardware Platform Evaluation Metric OLAP-C (Ours) ROMA (Prev. SOTA) Best Static Gain vs. ROMA
Intel Node (i7-10810U) Streaming AP (\(\text{sAP}\)) 0.287 0.206 0.154 +39.3%
Apple M3 (MacBook Air) Streaming AP (\(\text{sAP}\)) 0.391 0.314 0.301 +24.5%
Jetson Orin Nano Streaming AP (\(\text{sAP}\)) 0.408 0.403 0.399 +1.2%
Overall 71 Videos Avg Deadline Accuracy / sAP Superior Baseline Baseline +56% Accuracy / +39% sAP

Ablation Study

To isolate the benefit of fine-grained object-level modeling from feature representation richness, deadline prediction error was evaluated on the Argoverse-HD test split across multiple model variants. Baselines include FLAP (adapting the frame-level mean IoU decay model from ARISE) and OLAP-tiny (an object-level model retaining only execution latency and box IoU features):

Model Configuration Feature Granularity Effective Prediction Horizon RMSE Reduction vs. FLAP Note
FLAP (Frame-level Baseline) Frame-aggregated Short Horizon (\(\le 6\) frames / 0.2s) 0.0% (Baseline) Severely underestimates longer deadlines
OLAP-tiny Object-level (2 features) Medium Horizon 32.3% reduction Confirms the primary value of per-object modeling
OLAP (Full model) Object-level (10 features) Long Horizon (\(\le 15\) frames / 0.5s) 55.7% total reduction Additional 34.5% reduction over OLAP-tiny

Key Findings

  • Granularity Shift Drives Primary Accuracy Jump: Transitioning from frame-level aggregation (FLAP) to per-object modeling (OLAP-tiny) yields an immediate 32.3% average reduction in deadline estimation RMSE even with identical core features, demonstrating that frame-level averaging is the central error source in prior controllers.
  • Benefits Maximize Under Compute Scarcity: On the compute-constrained Intel Node where inference latencies exhibit a wide dispersion (49 ms to 3985 ms), OLAP-C achieves an 86.4% gain over Best Static and a 39.3% gain over ROMA. On Jetson Orin Nano where latency variance is narrow (54 ms to 161 ms), gains settle at 1.2%, proving that proactive deadline control is most vital under severe resource contention.
  • Priority Weights Provide Controllable Trade-offs: In the MOT17-13 pedestrian-crossing scenario, increasing the critical pedestrian weight to \(\omega_{\text{critical}} = 10\) directs the controller to favor low-latency detectors, preserving the pedestrian's tracking IoU above safety margins at the expense of marginal accuracy drops on distant, non-critical background objects.

Highlights & Insights

  • Closed-Form Inversion Eliminates Runtime Overhead: By modeling decay through exponential formulation and enforcing valid parameter ranges via Sigmoid and Softplus link functions, per-object deadlines are derived in closed form via \(C_q = \ln(\beta/\tau)/\gamma\). This keeps the runtime control overhead below 10% of detector inference latency.
  • Dual-Window Objective Replaces Heuristic Switching: Structuring utility into an immediate execution drop penalty (\(\text{AP}_{\text{drop}}\)) and a post-inference future expectation (\(\text{AP}_{\text{future}}\)) establishes a principled, predictive optimization target that avoids the myopia of reactive thresholding.
  • Decoupled Architecture Enables Broad Integration: The analytical per-object deadlines produced by OLAP serve as clean, model-agnostic physical timestamps, readily deployable as high-fidelity token inputs to autonomous planning stacks (such as UniAD or Chanakya) or as constraints for outer-loop DVFS governors.

Limitations & Future Work

  • Linear Motion and Short-Term Decay Assumptions: The model assumes that objects follow exponential IoU decay along smooth velocity vectors. Abrupt maneuvers, severe decelerations, or prolonged dynamic occlusions exceeding 0.5 s (15 frames) cause deadline predictions to become overly conservative.
  • Sensitivity to Detection Dropout: The cross-cycle matching step depends on consistent detections between successive cycles \(D_{k-1}\) and \(D_k\). False negatives in a single detection cycle sever object trajectories, temporarily dropping per-object deadline awareness.
  • Future Directions: Future extensions could incorporate lightweight Kalman filtering or graph-based relational modeling into the feature pipeline to track interacting agents, as well as extending the formulation to continuous streaming 3D LiDAR point clouds.
  • vs ROMA (WACV 2023): ROMA relies on frame-level predicted AP to drive model switching, failing to recognize when critical localized targets suffer severe degradation. OLAP-C resolves this with per-object deadline forecasting, delivering a 39% streaming AP improvement on edge platforms.
  • vs ARISE (MobiSys 2024): ARISE applies frame-average IoU decay for cloud offloading decisions; this paper shows that re-framing the identical core features at the per-object level (OLAP-tiny) cuts deadline error by 32.3%, while the full OLAP model cuts error by more than 55%.
  • vs Sela et al. (ECCV 2022): Sela et al. deploy heavy tree-based ensembles that require up to 13 GB of memory; OLAP relies on 10 explicit kinematic features with negligible memory and compute overhead, making it uniquely suited for constrained edge hardware.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Pioneers per-object exponential IoU decay modeling and closed-form deadline derivation for streaming detection control)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Validated across 71 video sequences from Argoverse-HD and MOT17 on three physical embedded devices spanning Intel CPU, Apple M3, and Jetson Orin Nano)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous mathematical formulation, clear problem motivation, and seamless alignment between theoretical claims and empirical data)
  • Value: ⭐⭐⭐⭐⭐ (Open-source implementation with minimal computational overhead addressing core latency-accuracy trade-offs in real-time robotic perception)