ODONet: Online Dynamic Offset Network for Visual Object Tracking¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/WhiteButterflies/ODONet
Area: Video Understanding
Keywords: visual object tracking, online dynamic offset learning, deformable attention, motion prior, adaptive receptive field
TL;DR¶
Addressing the issue that existing offset-based trackers operate offline on single frames and fail to adapt to continuous target appearance deformation and motion drift, ODONet injects historical bounding box motion prompts via two-way cross-attention at the encoding stage and dynamically fuses online propagated motion offsets with current offline offsets during inference, actively steering deformable attention toward an adaptive receptive field and setting state-of-the-art results on LaSOT (75.1% AUC) and GOT-10k (81.4% AO).
Background & Motivation¶
Visual object tracking faces significant challenges when targets undergo drastic deformation, rapid motion, or occlusion from background distractors. Recent trackers primarily adopt two lines of thought to enhance spatiotemporal representation: one expands the search region of interest (ROI) to the full frame or utilizes multi-frame feature concatenation to broaden the receptive field, but this spatial expansion inevitably captures irrelevant background clutter and distractor objects during feature extraction and fusion, exacerbating localization drift; the other reformulates tracking into an autoregressive trajectory sequence generation task, predicting current target coordinates from historical bounding boxes. However, such autoregressive paradigms typically incorporate historical coordinates only at the decoding stage, lacking fine-grained interactions between coordinate semantics and low-level visual features during encoding, which still leaves them reliant on indirect motion inference without explicit inter-frame geometric displacement modeling.
Meanwhile, in object detection and segmentation, offset learning techniques such as deformable convolution (DCN) and deformable attention have proven highly effective in adapting spatial sampling points to diverse object structures. Nonetheless, existing offset-based approaches generate sampling offsets completely offline within individual inference steps, optimizing spatial discriminability across generic categories rather than exploiting the spatiotemporal motion continuity of the same target in continuous video streams. Consequently, these offline offsets rupture temporal continuity, failing to provide forward-looking, motion-adaptive sampling when a target undergoes rapid acceleration or severe pose changes.
This paper addresses this gap by treating historical bounding boxes as explicit motion prompts, enabling the network to actively extrapolate and propagate dynamic offsets online during test-time inference. Core idea: convert historical bounding boxes into normalized motion prompts, inject them into visual features via a two-way Transformer during encoding to produce motion trend encodings, and during inference, combine the extrapolated online offsets with current offline visual offsets to dynamically steer deformable attention sampling grids across the search area.
Method¶
Overall Architecture¶
The overall architecture of ODONet comprises a multi-stage Transformer backbone (built upon Fast-iTPN-B), a Dynamic Visual-Motion Feature Interaction (DV-MFI) module, an Online Dynamic Offset Propagation (ODOP) module, stacked multi-layer feature interaction blocks, and an OSTrack-style bounding box prediction head. During tracking inference, the system takes \(N\) template images, the current search area image, and the previous frame's predicted target bounding box as inputs.
Templates and search area images are first fed through backbone Stage-0 to extract patch embeddings. The previous frame's normalized bounding box is treated as a motion prompt and fused with the search area visual tokens in DV-MFI via two-way cross-modal interaction, producing joint tokens carrying both appearance and motion priors. Next, in the ODOP module, the historical motion trend encoding is projected via a lightweight MLP and mapped through affine transformation into an online dense offset field, which is fused with an offline offset field predicted from current visual features. The resulting online dynamic offset guides deformable attention sampling in deeper backbone stages, dynamically tailoring the receptive field to the target's trajectory. Finally, the prediction head outputs target classification confidence and refined bounding box coordinates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Inputs: Search Image & Multiple Templates<br/>+ Previous Predicted Bounding Box"] --> B["Dynamic Visual-Motion Feature Interaction (DV-MFI)<br/>Two-Way Transformer Enriched with Motion Prompts"]
B --> C["Online Dynamic Offset Propagation (ODOP)<br/>Dynamic Injection of Online Extrapolation & Offline Observation"]
C --> D["Dense Multi-Task Motion Supervision & Head<br/>Adaptive Deformable Sampling & Box Regression"]
D --> E["Outputs: Current Target Bounding Box<br/>+ Dynamic Displacement Vector"]
Key Designs¶
1. Dynamic Visual-Motion Feature Interaction (DV-MFI): Early Cross-Modal Encoding Guided by Motion Prompts
When prior trackers incorporate historical coordinates globally, spatial scale misalignments frequently occur within locally cropped search regions. To eliminate this issue, DV-MFI normalizes the previous frame's bounding box relative to the current search area, yielding a compact geometric prior \(\tilde{P}_{t-1}^{pred} = [\tilde{x}_{t-1}, \tilde{y}_{t-1}, \tilde{w}_{t-1}, \tilde{h}_{t-1}] \in [0, 1]^4\). Drawing inspiration from the prompt mechanism in SAM, this bounding box is treated as a motion prompt and projected into a sparse motion trend encoding vector \(M = \phi(\tilde{P}_{t-1}^{pred})\) through a prompt embedding network \(\phi\).
To allow low-level visual features to perceive motion dynamics as early as possible, DV-MFI introduces a Two-Way Transformer (V&M Interaction Encoder). After Stage-0 visual feature extraction, the motion token \(M\) and the search area token \(X\) undergo sequential operations: self-attention on \(M\) to aggregate geometric priors, cross-attention from \(M\) (as query) to \(X\) (as key/value) to absorb appearance context, an MLP state update, and reverse cross-attention from \(X\) to the updated \(M\). This mechanism directly injects continuous temporal motion priors into low-level spatial features during encoding, preventing the semantic disconnect caused by postponing coordinate integration to the decoder.
2. Online Dynamic Offset Propagation (ODOP): Adaptive Feature Sampling via Blended Historical Extrapolation and Current Observation
To transform feature sampling from passive response into active anticipation, the ODOP module generates two complementary offset fields during test time: an online offset field \(\Delta ON_t\) extrapolated from historical motion, and an offline offset field \(\Delta OFF_t\) estimated from current visual features. The generation of \(\Delta ON_t\) utilizes the previous motion encoding \(M_{t-1}\); a lightweight MLP predicts the inter-frame box offset \(\Delta P_{t-1}^{odop} = \text{MLP}(M_{t-1}) \in [-1, 1]^4\), defining the extrapolated box \(\hat{P}_t^{odop} = \tilde{P}_{t-1}^{pred} + \Delta P_{t-1}^{odop}\). For any feature grid point \(p(i, j)\) inside \(\tilde{P}_{t-1}^{pred}\), its corresponding location \(p'(i_{target}, j_{target})\) inside \(\hat{P}_t^{odop}\) is determined by an affine transformation: $\(r_x = \frac{i - \tilde{x}_{t-1}}{\tilde{w}_{t-1}}, \quad r_y = \frac{j - \tilde{y}_{t-1}}{\tilde{h}_{t-1}}, \quad p' = (\hat{x}_t + r_x \hat{w}_t, \; \hat{y}_t + r_y \hat{h}_t)\)$ The displacement vector \(\Delta ON_t(i, j) = p' - p\) forms the dense online motion prior inside the target box, while regions outside are zero-padded. Concurrently, search feature \(X\) is processed via convolution, GELU, and \(\tanh\) activation to predict the global offline offset field \(\Delta OFF_t\). Through the Online Offset Injection (OOI) strategy, both fields are linearly fused: $\(\Delta OND_t = w_{on} \cdot \Delta ON_t + (1 - w_{on}) \cdot \Delta OFF_t\)$ where the hyperparameter \(w_{on} = 0.5\). The fused online dynamic offset \(\Delta OND_t\) directly modulates the regular grid coordinates of deformable attention, where keys and values are sampled via bilinear interpolation, steering the receptive field to dynamically track target translation and scale variations.
3. Dense Multi-Task Motion Supervision: Joint Directional and Magnitude Constraints for End-to-End Learning
Because conventional offline offsets are typically optimized solely via indirect task gradients, they lack explicit physical displacement constraints, often leading to unconstrained and noisy sampling patterns. To ensure that both the extrapolated box displacement \(\Delta P_t^{odop}\) and the offline field \(\Delta OFF_t\) capture genuine physical motion, authors introduce multi-task dense supervision. For global target displacement, an L1 loss is imposed between the predicted motion vector and ground-truth inter-frame displacement \(\Delta P_t^{gt} = P_t^{gt} - \tilde{P}_{t-1}^{gt}\): $\(\mathcal{L}_{motion} = \|\Delta P_t^{odop} - \Delta P_t^{gt}\|_1\)$ For the offline offset field \(\Delta OFF_t\), ground-truth bounding box annotations are fed into the affine formulation to construct a dense pseudo-ground-truth offset field \(\Delta OFF_t^{gt} = \text{OnlineOffsetGeneration}(\tilde{P}_{t-1}^{gt}, P_t^{gt})\). Both magnitude and orientation are constrained via L1 and cosine similarity losses: $\(\mathcal{L}_{offset-l1} = \|\Delta OFF_t - \Delta OFF_t^{gt}\|_1\)$ $\(\mathcal{L}_{offset-cos} = 1 - \frac{\langle \Delta OFF_t, \Delta OFF_t^{gt} \rangle}{\|\Delta OFF_t\| \|\Delta OFF_t^{gt}\|}\)$ These two losses are computed exclusively within the region covered by \(\tilde{P}_{t-1}^{gt}\) inside the search frame, ensuring that learned offsets align strictly in both movement scale and vector orientation with real physical displacement.
A Worked Example¶
Consider tracking a fast-moving drone video where a vehicle accelerates toward the top-right corner. At frame \(t-1\), the vehicle is tracked with normalized box \(\tilde{P}_{t-1}^{pred}\). At frame \(t\): 1. The prompt encoder maps \(\tilde{P}_{t-1}^{pred}\) into motion vector \(M\), and through DV-MFI two-way cross-attention, spatial tokens along the vehicle's forward trajectory in search feature \(X\) receive intensified activation; 2. ODOP's MLP predicts the box displacement \(\Delta P_{t-1}^{odop} = [+0.08, -0.05, +0.01, +0.01]\), indicating a right-upward shift with slight scale expansion; 3. The affine transformation maps internal grid points to their predicted top-right coordinates, creating the dense online offset field \(\Delta ON_t\); 4. Concurrently, offline offset prediction extracts \(\Delta OFF_t\) from current visual features, and OOI combines them with \(w_{on} = 0.5\) to produce \(\Delta OND_t\); 5. The deformable attention sampling points \(p_q\) shift toward the predicted vehicle region in the top-right quadrant, avoiding distractors in the bottom-left background; 6. The prediction head aggregates the motion-focused features and outputs high-precision coordinates with zero drift.
Loss & Training¶
The framework is optimized end-to-end using a weighted multi-task objective: $\(\mathcal{L} = \lambda_f \mathcal{L}_{focal} + \lambda_b \mathcal{L}_{box-l1} + \lambda_g \mathcal{L}_{giou} + \mathcal{L}_{motion} + \mathcal{L}_{offset-l1} + \mathcal{L}_{offset-cos}\)$ Loss weights are set to \(\lambda_b = 5\), \(\lambda_g = 2\), \(\lambda_f = 1\), with all motion and offset supervision weights set to 1. The backbone uses Fast-iTPN-B with 5 templates. Training is conducted on LaSOT, TrackingNet, GOT-10k, COCO, and VastTrack for 300 epochs (100 epochs when training solely on GOT-10k under the official one-shot benchmark protocol). The AdamW optimizer is employed with a learning rate of \(4 \times 10^{-5}\) for the backbone, \(4 \times 10^{-4}\) for other modules, and a weight decay of \(10^{-4}\). Training is executed across 4 RTX 3090 GPUs with a batch size of 128.
Key Experimental Results¶
Main Results¶
On major long-term and large-scale benchmarks (LaSOT, LaSOText, TrackingNet) as well as the zero-overlap evaluation benchmark (GOT-10k), ODONet demonstrates consistent gains over competitive SOTA trackers.
| Method | LaSOT (AUC) | LaSOT (PNorm) | LaSOText (AUC) | TrackingNet (AUC) | TrackingNet (PNorm) | GOT-10k (AO)* | GOT-10k (SR0.5)* |
|---|---|---|---|---|---|---|---|
| ODONet-B384 (Ours) | 75.1 | 84.5 | 53.1 | 87.4 | 91.6 | 81.4 | 90.0 |
| ODONet-B224 (Ours) | 74.6 | 84.3 | 54.3 | 86.3 | 90.8 | 78.9 | 89.6 |
| SPMTrack-B378 [CVPR'25] | 74.9 | 84.0 | - | 86.1 | 90.2 | 76.5 | 85.9 |
| SUTrack-B384 [AAAI'25] | 74.4 | 83.9 | 52.9 | 86.5 | 90.7 | 79.3 | 88.0 |
| ARTrackV2-L384 [CVPR'24] | 73.6 | 82.8 | 53.4 | 86.1 | 90.4 | 79.5 | 87.8 |
| ODTrack-B384 [AAAI'24] | 73.2 | 83.2 | 52.4 | 85.1 | 90.1 | 77.0 | 87.9 |
| ARTrack-L384 [CVPR'23] | 73.1 | 82.2 | 52.8 | 85.6 | 89.6 | 78.5 | 87.4 |
| SeqTrack-L384 [CVPR'23] | 72.5 | 81.5 | 50.7 | 85.5 | 89.8 | 74.8 | 81.9 |
| OSTrack-256 [ECCV'22] | 69.1 | 78.7 | 47.4 | 83.1 | 87.8 | 71.0 | 80.4 |
On additional specialized benchmarks, ODONet achieves 63.1% AUC on text-guided TNL2K (surpassing MambaLCT at 58.5%), 71.4% AUC on aerial benchmark UAV123, and 61.6 EAO on VOT2020 when paired with Alpha-Refine.
Ablation Study¶
Ablation investigations were conducted on the GOT-10k benchmark to isolate component contributions and evaluate sensitivity to the online fusion parameter \(w_{on}\).
| Experiment | Configuration / Parameter | GOT-10k (AO) | Params (M) | MACs (G) | Note |
|---|---|---|---|---|---|
| Module Ablation | Baseline (Fixed Receptive Field) | 77.1 | 81.3 | 37.1 | Vanilla ViT with static grid sampling |
| Module Ablation | + DV-MFI | 77.6 (+0.5) | 89.7 | 37.5 | Motion prompts injected into visual features |
| Module Ablation | + ODOP | 78.0 (+0.9) | 82.4 | 37.3 | Deformable attention with offsets |
| Module Ablation | DV-MFI + ODOP (Full ODONet) | 78.9 (+1.8) | 90.8 | 37.7 | +11.7% Params, +1.6% MACs |
| Ratio Analysis | \(w_{on} = 0.25\) | 78.2 | 90.8 | 37.7 | Insufficient historical online prior |
| Ratio Analysis | \(w_{on} = 0.50\) (Default) | 78.9 | 90.8 | 37.7 | Optimal balance of motion and observation |
| Ratio Analysis | \(w_{on} = 0.75\) | 77.9 | 90.8 | 37.7 | Over-reliance on extrapolated motion |
| Ratio Analysis | \(w_{on} = 1.00\) | 78.5 | 90.8 | 37.7 | Completely discarding offline observation |
Key Findings¶
- High Complementarity of Motion Prompts and Dynamic Sampling: Introducing DV-MFI alone improves AO by 0.5%, and ODOP alone yields a 0.9% gain; combining both drives an overall 1.8% jump (77.1% \(\to\) 78.9%), demonstrating that feature-level motion injection and spatial sampling guidance reinforce each other.
- Inverted U-Curve of the Online Ratio \(w_{on}\): Setting \(w_{on} = 0.5\) strikes the ideal balance. Lower ratios underutilize temporal inertia, while setting \(w_{on} > 0.5\) leads to performance fluctuations due to error propagation from previous tracking inaccuracies.
- Superior Cross-Category Generalization: Under GOT-10k's zero-overlap protocol, ODONet-B384 reaches 81.4% AO (+2.1% over SUTrack-B384), confirming that the network learns transferable geometric deformation and motion kinematics rather than semantic over-fitting.
Highlights & Insights¶
- From Passive Correlation to Active Motion Extrapolation: While typical attention-based trackers passively compute cross-attention over entire search patches, ODONet is the first tracker to propagate dynamic offsets online during test time, granting the model proactive trajectory awareness.
- Unsupervised Failure Diagnosis via Offset Gram Matrix: Drawing inspiration from image style transfer, authors compute the Gram matrix of \(\Delta OND_t\) across horizontal (\(G[0, 0]\)) and vertical (\(G[1, 1]\)) dimensions. Abnormal sharp peaks in the Gram matrix consistently coincide with catastrophic drops in frame-wise \(P_{norm}\), serving as an effective unsupervised tracking health indicator.
- Early Warning Before Target Drift: Visual diagnosis reveals that when distractor objects enter the search area, traditional confidence heatmaps exhibit ambiguous multi-peak responses; in contrast, ODONet's dynamic offset vector field \(\Delta OND_t\) explicitly shifts its direction toward the distractor one frame earlier, providing an early warning of tracking drift.
Limitations & Future Work¶
- Rigid-Body and Affine Assumptions: Online offset generation relies on the hypothesis that relative coordinate topology within the bounding box remains invariant over brief temporal intervals. This assumption degrades under violent non-rigid twisting, severe self-occlusion, or out-of-plane 3D rotations.
- Dual Computation Overhead: Although the lightweight MLP introduces merely 1.6% additional MACs, calculating per-point coordinate transformations and executing deformable bilinear interpolation across dense feature maps can increase memory bandwidth pressure on resource-constrained edge chips.
- Future Directions: Integrating explicit optical flow constraints or monocular depth cues into the online propagation loop could enable full 3D spatial dynamic offsets; additionally, Gram matrix failure spikes could trigger adaptive template refresh mechanisms.
Related Work & Insights¶
- vs OSTrack [ECCV 2022] / MixFormer [CVPR 2022]: OSTrack and MixFormer perform global feature correlation across static grids, exposing feature representations to severe background clutter; ODONet preserves one-stream efficiency while dynamically steering attention sampling toward the moving object.
- vs DeformDETR [ICLR 2020] / D-TransT [Electronics 2022]: Prior deformable attention trackers generate offsets solely from offline static features without inter-frame motion awareness; ODONet propagates dynamic offsets online using historical trajectory prompts, ensuring continuous spatiotemporal adaptation.
- vs ARTrack / ARTrackV2 [CVPR 2023/2024]: Autoregressive trackers pass historical coordinate tokens into the decoder, missing the chance to adapt spatial receptive fields at early feature extraction stages; ODONet injects motion prompts right at Stage-0 via two-way cross-attention, deeply aligning spatial and temporal representations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneers online dynamic offset propagation during inference for visual tracking, breaking the static offset paradigm]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 7 major benchmarks, detailed ablation of online ratios, and an insightful Gram matrix diagnostic analysis]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous problem formulation, transparent architectural description, and clear mathematical derivations]
- Value: ⭐⭐⭐⭐⭐ [Offers both SOTA tracking accuracy and a novel diagnostic tool, providing valuable paradigms for future dynamic attention trackers]