Skip to content

GAP-Track: Bridging the Resolution Gap for Cross-Resolution RGBT Tracking

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/shijiahahaha/GAPTrack
Area: Object Detection
Keywords: RGBT tracking, cross-resolution tracking, knowledge distillation, feature reconstruction, polar-geometric loss

TL;DR

Addressing performance degradation on edge devices constrained by low compute and narrow bandwidth, GAP-Track combines training-time masked residual reconstruction, polar-geometric sensitivity loss, and hierarchical knowledge distillation to enable high-precision cross-resolution RGBT tracking with zero inference overhead.

Background & Motivation

In all-weather autonomous systems and video surveillance infrastructures, RGB-Thermal (RGBT) multimodal visual tracking plays an indispensable role due to the complementary characteristics of visible and infrared spectra. Visible light sensors deliver dense textural cues and sharp object contours under optimal illumination, but fail in poor lighting or dense fog. Conversely, thermal infrared sensors record emitted thermal radiation, highlighting object footprints across adverse nighttime or foggy weather. Nonetheless, when deploying RGBT tracking systems onto real-world edge hardware platforms and airborne drones, limited computing power and tight transmission bandwidth dictate that incoming video streams must be downsampled. Downsampling full-resolution inputs to low or extreme quarter resolutions (e.g., \(1/2\) or \(1/4\) scale) slashes bandwidth consumption down to \(6.25\%\) and dramatically reduces memory footprint, serving as a prerequisite for scalable edge deployment.

Despite these efficiency gains, modern top-tier RGBT trackers suffer severe performance collapse as the input resolution drops. This degradation stems from two fundamental tensions: first, spatial ambiguity intensifies because aggressive downsampling eliminates high-frequency geometric textures and boundary cues, reducing medium-sized targets into blurry blobs that span merely several feature pixels. Second, cross-modal correspondence deteriorates, as spatial alignment between RGB and thermal channels becomes severely distorted at coarse granularities, impeding discriminative multimodal fusion. Furthermore, traditional bounding box regression objectives (such as smooth L1, L2, or standard IoU loss) suffer from vanishing gradients and optimization "dead zones" near the target center when visual cues are extremely sparse, leading to unrecoverable center-point drift and poor scale estimation.

Directly forcing low-resolution student features to mimic full-resolution teacher features via standard distillation is prone to failure due to the vast resolution disparity. The angle of attack in this paper is that robust low-resolution tracking demands joint innovation across training-time representation learning and geometric optimization sensitivity, while completely discarding auxiliary branches at test time. Core idea: propose GAP-Track, a cross-resolution tracking framework that forces low-resolution features to recover high-frequency structural details via Auxiliary Generative Reconstruction (AGR), resolves center drift and gradient vanishing via Polar-Geometric Sensitivity (PGS) loss, and stabilizes extreme downsampling optimization via Hierarchical Knowledge Distillation (HKD).

Method

Overall Architecture

GAP-Track adopts an end-to-end teacher-student distillation architecture. During training, high-resolution template and search region patches are downsampled into multiple low-resolution inputs and fed into the student backbone. To bridge the resolution discrepancy, the framework introduces three cooperative modules that operate strictly as training-time auxiliary supervisors: the Auxiliary Generative Reconstruction (AGR) module uses random masking and a lightweight residual generator to reconstruct teacher-level feature details; the Hierarchical Knowledge Distillation (HKD) strategy adaptively introduces an intermediate half-resolution teacher to smooth out optimization trajectories under extreme \(1/4\) resolution; and the Polar-Geometric Sensitivity (PGS) loss supervises the tracking head by decoupling location regression into radial and angular components. During inference, all auxiliary components—including the teacher model, generative branch, and distillation losses—are discarded, leaving only the student backbone and tracking head running at peak real-time efficiency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Low-Res Inputs<br/>RGB & TIR Patches"] --> B["Student Backbone Feature Extraction<br/>Fs Low-Res Feature Map"]
    B --> C["Auxiliary Generative Reconstruction<br/>Bilinear Upsampling + Random Masking + Residual Generator"]
    B --> D["Hierarchical Knowledge Distillation<br/>Multi-Stage Alignment via Intermediate Teacher"]
    B --> E["Tracking Prediction Head<br/>Predicts Target Center & Box Bounds"]
    E --> F["Polar-Geometric Sensitivity Loss<br/>Polar Displaced Location + Variance-Aware Scale Loss"]

Key Designs

1. Auxiliary Generative Reconstruction: Forcing Backbone Structural Encoding via Masked Residual Learning

Conventional feature distillation forces low-resolution features to mimic high-resolution representations directly, causing the student to memorize blurry interpolated values rather than recovering genuine high-frequency edges. To overcome this limitation, GAP-Track incorporates an Auxiliary Generative Reconstruction (AGR) task. Given the low-resolution feature map \(F_s \in \mathbb{R}^{C \times H_s \times W_s}\) extracted by the student network, spatial dimensions are first aligned via bilinear upsampling to yield \(F_{up}\). A spatial binary mask \(M \in \{0, 1\}^{H \times W}\) is then applied to generate masked features \(F_{masked} = F_{up} \odot M\). To maintain training stability and prevent semantic corruption, a lightweight CNN-based residual generator \(G\) predicts only the missing high-frequency residual: $\(F_{rec} = F_{up} + G(F_{masked})\)$ A content loss supervises \(F_{rec}\) against high-resolution teacher features \(F_t\). The random masking prevents shortcut memorization, forcing the student backbone to infer structural context from surrounding cues, while the residual connection allows the generator to focus on sharp object boundaries. Crucially, the generator \(G\) is discarded at inference time, delivering refined feature representations at zero extra computational cost.

2. Polar-Geometric Sensitivity Loss: Polar Coordinate Decoupling and Second-Order Sensitivity to Eliminate Center Drift

Under extreme \(64 \times 64\) downsampling, standard Cartesian L2 or IoU losses suffer from a wide "gradient dead zone" near the target center where gradients vanish linearly with diminishing offset. This leads to severe center-point drift for tiny blurred targets. To address this issue, the Polar-Geometric Sensitivity (PGS) loss decomposes boundary box regression into a scale-sensitive loss \(\mathcal{L}_S\) and a polar-location sensitive loss \(\mathcal{L}_{PLS}\): $\(\mathcal{L}_{PGS} = \mathcal{L}_S \cdot \mathcal{L}_{PLS}\)$ For the location component, the predicted center \(c_p=(x_p, y_p)\) and ground-truth center \(c_{gt}=(x_{gt}, y_{gt})\) are converted into polar coordinates \((d_i, \theta_i)\) relative to the search region center \((x_c, y_c)\). The location loss explicitly multiplies radial sensitivity by angular sensitivity: $\(\mathcal{L}_{PLS} = \left( 1 - \frac{\min(d_p, d_{gt})}{\max(d_p, d_{gt})} \right) \cdot \frac{4}{\pi^2}(\theta_p \theta_{gt})^2\)$ Because the polar coordinate transformation possesses a non-linear Jacobian matrix, its derivative with respect to spatial coordinates contains a term proportional to \(1/d\). As distance \(d \to 0\), this term counteracts diminishing spatial offsets, producing a steep funnel-shaped optimization landscape that eliminates gradient dead zones. For the scale component, a variance-aware weighting factor \(w\) modulates the IoU calculation: $\(w = \frac{\min(|A_p|, |A_{gt}|) + \text{Var}(|A_p|, |A_{gt}|)}{\max(|A_p|, |A_{gt}|) \cdot \text{Var}(|A_p|, |A_{gt}|)}\)$ This weighting heavily penalizes severe scale discrepancies, preventing the bounding box from settling into sub-optimal local minima.

3. Hierarchical Knowledge Distillation: Bridging Cross-Scale Knowledge Discrepancies with Intermediate Teachers

When the input resolution is sharply reduced from full scale (\(256 \times 256\)) to quarter scale (\(64 \times 64\)), the substantial gap in receptive field and feature abstraction causes direct cross-scale distillation to destabilize. GAP-Track implements a Hierarchical Knowledge Distillation (HKD) strategy. For half-resolution students, knowledge is transferred directly from the full-resolution teacher. For quarter-resolution students, a high-performing half-resolution model is introduced as an intermediate teacher. Using learnable projection layers \(\phi(\cdot)\) (comprising \(1 \times 1\) convolutions and bilinear interpolation), multi-stage backbone feature maps are dynamically aligned: $\(\mathcal{L}_{HKD} = \sum_{i} \alpha_i F_{\mathcal{T}}^i \phi(F_{\mathcal{S}}^i)\)$ This stepping-stone design lets the low-resolution student first master multi-modal alignment at an intermediate scale before tackling severe structural degradation, stabilizing convergence under extreme downsampling.

Loss & Training

The joint optimization objective of GAP-Track weights the Auxiliary Generative Reconstruction loss \(\mathcal{L}_{AGR}\), Polar-Geometric Sensitivity loss \(\mathcal{L}_{PGS}\), and Hierarchical Knowledge Distillation loss \(\mathcal{L}_{HKD}\): $\(\mathcal{L}_{total} = \lambda_{AGR}\mathcal{L}_{AGR} + \lambda_{PGS}\mathcal{L}_{PGS} + \lambda_{HKD}\mathcal{L}_{HKD}\)$ Implemented in PyTorch, the network is trained on a single NVIDIA RTX 3090 GPU. The backbone is initialized from DropMAE pre-trained weights. Training employs the AdamW optimizer with an initial learning rate of \(1 \times 10^{-4}\), weight decay of \(1 \times 10^{-4}\), batch size of 32, and 60,000 image samples per epoch across 30 epochs.

Key Experimental Results

Main Results

On the two premier large-scale RGBT tracking benchmarks LasHeR and RGBT234, GAP-Track was evaluated across full resolution (\(256 \times 256\)), half resolution (\(128 \times 128\)), and quarter resolution (\(64 \times 64\)).

Dataset Resolution Setting Metric GAP-Track (Ours) CKD (Prev. SOTA / Baseline) Gain / Difference
LasHeR Full Res (\(256 \times 256\)) PR / SR / NPR 71.8 / 57.1 / 67.9 73.2 / 58.1 / 69.3 Competitive tier-one baseline
LasHeR Half Res (\(128 \times 128\)) PR / SR / NPR 67.6 / 52.9 / 63.3 65.7 / 51.5 / 61.6 +1.9% PR / +1.4% SR / +1.7% NPR
LasHeR Quart. Res (\(64 \times 64\)) PR / SR / NPR 57.2 / 43.7 / 52.7 44.1 / 35.3 / 39.4 +13.1% PR / +8.4% SR / +13.3% NPR
RGBT234 Full Res (\(256 \times 256\)) PR / SR 87.4 / 64.8 90.0 / 67.4 Preserves solid full-resolution foundation
RGBT234 Half Res (\(128 \times 128\)) PR / SR 84.2 / 62.5 83.3 / 60.7 +0.9% PR / +1.8% SR
RGBT234 Quart. Res (\(64 \times 64\)) PR / SR 75.5 / 54.5 68.2 / 47.8 +7.3% PR / +6.7% SR

In terms of computational complexity, GAP-Track retains the exact same parameter size (183.43M) and FLOPs as the baseline (only 11.70G FLOPs at quarter resolution, a 90% reduction compared to 117.01G at full resolution). Furthermore, when integrating GAP-Track's auxiliary supervisors into SUTrack (AAAI 2025) under half-resolution, it yields consistent improvements across three distinct benchmarks and modalities: LasHeR (RGBT, \(+0.7\%\) SR), LaSOT (pure RGB, \(+1.1\%\) AUC), and VisEvent (RGB-Event, \(+0.9\%\) AUC).

Ablation Study

A step-by-step ablation on the LasHeR dataset demonstrates the individual and combined impact of the proposed components under half and quarter resolutions:

Config AGR HKD PGS Half Res (PR/SR/NPR) Quart. Res (PR/SR/NPR) Note
Baseline - - - 65.7 / 51.5 / 61.6 44.1 / 35.3 / 39.4 Standard student collapses at 1/4 res
+ AGR ✓ - - 66.8 / 52.3 / 62.4 49.7 / 39.2 / 44.1 High-frequency detail recovery (+5.6% PR)
+ AGR + PGS ✓ - ✓ 67.6 / 52.9 / 63.3 54.1 / 41.8 / 48.8 Second-order gradient suppresses drift (+4.4% PR)
Full Model (+HKD) ✓ ✓ ✓ 67.6 / 52.9 / 63.3 57.2 / 43.7 / 52.7 Intermediate teacher bridges scale gap (+3.1% PR)

Key Findings

  • AGR recovers semantic discriminability under heavy spatial decay: Adding AGR boosts 1/4 resolution PR by \(+5.6\%\). Grad-CAM activations show it refocuses scattered cross-modal attention directly onto true target centers.
  • PGS loss suppresses localization drift via second-order geometric sensitivity: PGS contributes an additional \(+4.4\%\) PR and \(+2.6\%\) SR at quarter resolution. The non-linear Jacobian of polar coordinates maintains sharp gradient supervision where Cartesian L2 loss encounters a flat dead zone.
  • HKD serves as an essential stepping stone across extreme resolution gaps: Attempting direct distillation across a \(4\times\) spatial downscaling factor creates optimization friction; introducing a \(1/2\)-resolution intermediate teacher provides a smooth trajectory that yields a \(+3.1\%\) PR improvement.

Highlights & Insights

  • Zero-overhead inference through training-time auxiliary supervision: By relegating generative feature reconstruction and polar transformation calculations strictly to the training phase, the model achieves substantial performance gains without adding any parameters or FLOPs during inference.
  • Tackling gradient vanishing with polar coordinate transforms: Instead of designing increasingly complex box IoU variants in Cartesian space, reformulating target regression in polar coordinates harnesses the natural \(1/d\) derivative singularity to eliminate center-point gradient starvation.
  • Universal plug-and-play capability across vision modalities: The framework demonstrates solid model-agnostic generalizability, bringing consistent accuracy boosts not only to RGBT tracking but also to pure RGB tracking and neuromorphic RGB-Event tracking.

Limitations & Future Work

  • Lack of temporal dynamic compensation: GAP-Track focuses primarily on single-frame spatial geometry and cross-modal resolution alignment. Under continuous occlusions or abrupt thermal signature shifts, leveraging historical multi-frame memory caches could help recover lost spatial cues.
  • Absence of dynamic resolution scaling: The current framework uses fixed downsampling factors (\(1/2\) or \(1/4\)). An adaptive mechanism that dynamically shifts input resolution based on real-time edge battery levels or network congestion remains an interesting direction for future exploration.
  • vs CKD (Coupled Knowledge Distillation, ACM MM 2024): CKD concentrates on bridging the multimodal gap between template and search regions at standard resolution. However, it lacks high-frequency reconstruction under severe downsampling; GAP-Track outperforms CKD by over 13% PR at 1/4 resolution with identical inference cost.
  • vs DropMAE / ViTKD: While conventional masked autoencoders are geared toward self-supervised pretraining, GAP-Track repurposes masked reconstruction as an auxiliary super-resolution bridge, specifically targeting task-critical high-frequency residuals.

Rating

  • Novelty: ⭐⭐⭐⭐☆ The polar-geometric formulation effectively solves center-point gradient vanishing, and the masked residual distillation bridge is well-motivated.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across LasHeR, RGBT234, LaSOT, and VisEvent, backed by thorough FLOPs profiling and ablation studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear narrative progression, rigorous mathematical formulation, and convincing loss landscape visualizations.
  • Value: ⭐⭐⭐⭐⭐ Highly practical paradigm for deploying real-time multimodal trackers on resource-constrained edge platforms.