HieDG: A Hierarchical Discrete Geometry-Guided Framework for Multi-Animal Tracking¶

Conference: ECCV 2026
arXiv: 2607.00494
Code: None
Area: Video Understanding
Keywords: Multi-Animal Tracking, Discrete Geometric Representation, Residual Vector Quantization, Query-based Multi-Object Tracking, Identity Association

TL;DR¶

HieDG discretizes unstable continuous geometric signals (position, scale, velocity) in multi-animal tracking into structured tokens via a two-level residual codebook, which are then fused with visual features and injected into a query-based tracker. It achieves SOTA association performance (with significant improvements in HOTA, AssA, and IDF1) on three animal tracking benchmarks (AnimalTrack, BFT, and BuckTales), and demonstrates generalizability on DanceTrack and SportsMOT.

Background & Motivation¶

Multi-animal tracking (MAT) is a fundamental task in wildlife monitoring and behavioral analysis. However, highly similar appearances of identical species, dense groups, and irregular motion trajectories make identity association extremely challenging. Existing methods are divided into two main paradigms: heuristic methods (e.g., SORT, ByteTrack, OC-SORT) rely on hand-crafted geometric matching rules (Kalman filtering + Hungarian algorithm), which perform explicit geometric modeling but suffer from poor scenario adaptability due to the lack of end-to-end optimization; query-based methods (e.g., MOTR, MOTIP) model end-to-end identity association through cross-attention but heavily rely on appearance embeddings, which are highly indistinguishable in animal scenarios, leading to error-prone associations.

A natural intuition is to introduce geometric cues into query-based trackers to combine the strengths of both paradigms. However, geometric modeling under attention architectures is non-trivial: camera jitters, target trembles, and environmental noise introduce subtle but persistent fluctuations in geometric signals. Although small in the raw analog space, these fluctuations can cause disproportionate shifts in cross-attention weight distributions after embedding, thereby disrupting the stability of identity matching. Experiments also confirm that directly utilizing continuous geometric representations (HieDG*) yields association performance even lower than the pure appearance-based baseline.

Inspired by quantization principles in signal processing—which map noisy analog signals to stable discrete states to suppress perturbations—this study revisits geometric modeling from the perspective of representation learning: mapping fluctuating continuous geometric signals into discrete tokens composed of a finite number of learnable codewords via hierarchical residual vector quantization (VQ), thereby constraining embedding variance while preserving the structural information required for identity association. The key insight is: to transform geometric modeling from a continuous regression problem into a discrete representation learning problem, eliminating the amplification effect of minor perturbations on attention weights through quantization.

Method¶

Overall Architecture¶

HieDG is built on the Deformable DETR detector, adding a geometric branch on top of the MOTIP baseline. The overall pipeline is as follows: the input video frames extract image features through a CNN backbone + Transformer encoder, and the Deformable DETR decoder outputs detection boxes and object-level visual embeddings \(f_t^i\). Meanwhile, normalized geometric vectors (position, scale, polar velocity) are constructed from the detection results, projected through MLPs, and then quantized using independent two-level residual codebooks to obtain discrete geometric embeddings \(\tilde{z}_t^i\). Finally, the visual embeddings, discrete geometric embeddings, and identity embeddings are concatenated and fed into the ID decoder to interact with historical trajectory features via cross-attention, with the classification head outputting identity predictions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video Frames"] --> B["Deformable DETR<br/>Detection + Visual Feature Extraction"]
    B --> C["Geometric Vector Construction<br/>Position/Scale/Polar Velocity"]
    B --> D["Visual Embedding f"]
    C --> E["Hierarchical Residual Quantization<br/>Two-level Codebook Coarse-to-Fine Discretization"]
    E --> F["Discrete Geometric Embedding z̃"]
    D --> G["Feature Concatenation<br/>Visual + Geometric + Identity"]
    F --> G
    H["Historical Trajectory Features"] --> I["ID Decoder<br/>Cross-Attention"]
    G --> I
    I --> J["Identity Classification Head<br/>Trajectory Association"]

Key Designs¶

1. Hierarchical Residual Geometric Quantization: Transforming continuous geometric signals into stable discrete tokens with two-level codebooks

In multi-animal scenarios, the position, scale, and velocity of detection boxes undergo continuous, minor fluctuations due to camera shake and irregular motion. These fluctuations are amplified by the attention mechanism in continuous embedding spaces—minor shifts of the same target in adjacent frames can dramatically alter the attention weight distribution, leading to identity mismatch. The core solution of HieDG is to independently perform coarse-to-fine discretization for each geometric component (position \(g_p\), scale \(g_s\), velocity \(g_v\)) using a two-level residual codebook.

Specifically, each geometric component is first projected into a \(D=64\) dimensional latent space via an MLP to obtain \(z_g\). A nearest neighbor search is then performed on the first-level codebook \(E_g^{(1)}\) (\(K=64\) codewords) to obtain the coarse codeword \(q_{g,1}\). After computing the residual \(r_g = z_g - q_{g,1}\), the second-level codebook \(E_g^{(2)}\) is used to quantize the residual, obtaining \(q_{g,2}\). The final discrete embedding is \(\tilde{z}_g = q_{g,1} + q_{g,2}\). The key to the two-level design is that the coarse codebook captures the macroscopic structure of the geometric status (e.g., "the target is in the upper left region of the screen"), while the fine codebook models the residual variations after coarse quantization (e.g., "fine-tuning a few pixels inside the upper left region"). This residual formulation significantly lightens the codeword burden of each stage by only requiring the second stage to model the offsets rather than absolute values. The discrete embeddings of the three components are concatenated and linearly projected to \(d\) dimensions to align with the visual features.

Ablation studies demonstrate that fixed-bin quantization (FixVQ) offers only marginal improvement, single-level VQ ranks second, and two-level residual VQ performs the best (HOTA 71.3 vs. 70.5 for single-level vs. 69.2 without quantization on BFT). A three-level VQ yields only a tiny improvement of 0.2 HOTA while introducing an additional 49K parameters and 98K FLOPs. Performance steadily improves as the codebook dimension increases from 16 to 32 to 64, saturating at 128.

2. Polar Velocity Representation: Decoupling motion magnitude and direction to cope with irregular motion

Animal motion often contains irregular patterns such as sudden stops and sharp turns. Cartesian velocity \((v_x, v_y)\) entangles magnitude and direction—different velocities in the same direction or the same velocity in different directions map to far-apart points in the embedding space, weakening the embedding compactness of the same identity's trajectory. HieDG converts the velocity obtained from inter-frame differences from Cartesian coordinates to polar coordinates \((\rho, \theta)\), where \(\rho\) represents the motion magnitude and \(\theta\) represents the motion direction, physically decoupling the two. Ablations show that polar velocity yields stable improvements in AssA and IDF1 compared to Cartesian velocity, validating the effectiveness of decoupling in modeling irregular motions.

3. KNN Velocity Estimation during Inference: Resolving missing velocity when inter-frame correspondence is absent

During training, velocity can be directly computed via the difference between detection boxes in adjacent frames. However, during inference, current detections have not yet been associated with historical trajectories, leaving the velocity unavailable. For each current-frame detection, HieDG retrieves the \(k\) nearest neighbors in spatial coordinates from the historical trajectory state set \(\mathcal{H}\), estimating its velocity via normalized weighted pooling:

\[\hat{\mathbf{v}}_t^i = \sum_{j \in \mathcal{N}_k} w_{ij} \mathbf{v}_j, \quad w_{ij} \propto \exp\left(-\frac{\|\mathbf{c}_t^i - \mathbf{c}_j\|_2^2}{\sigma_s^2}\right) \cdot \exp\left(\frac{\cos(\Delta\phi_{ij})}{\sigma_d}\right)\]

The weights are determined by the product of two kernel functions: the spatial proximity kernel encourages historical trajectories with closer physical distance to contribute more, and the directional consistency kernel encourages trajectories aligned in motion direction to contribute more. This lightweight KNN estimation introduces no additional learnable parameters and is fully compatible with the quantization module—the estimated Cartesian velocity is similarly converted to polar coordinates before entering the quantization pipeline. A dropout of 0.3 is applied to velocity embeddings during training to enhance robustness to estimation noise.

A Concrete Example: How Quantization Stabilizes Attention¶

Assume a deer has a normalized position of \((0.32, 0.45)\) at frame \(t\), which shifts to \((0.33, 0.44)\) at frame \(t+1\) due to minor camera jitters. If continuous geometric embeddings are used directly, these two close positions might be mapped by the MLP to regions with highly disparate attention weights in the embedding space, causing the model to misclassify them as different identities. In HieDG, however, \((0.32, 0.45)\) is quantized into the nearest codeword (e.g., codeword #23, corresponding to "upper-left region of the view") by the first-level codebook, and the residual \((-0.01, +0.01)\) is quantized into a fine-tuning codeword (e.g., codeword #7) by the second-level codebook. Similarly, \((0.33, 0.44)\) is also mapped to codeword #23 + codeword #7. Both frames generate exactly the same discrete token, keeping the attention weights stable and preventing identity association from being disrupted by camera jitters. This illustrates the core value of quantization in "compressing fluctuating signals into the same discrete state."

Loss & Training¶

The total loss is a weighted sum of three terms: \(\mathcal{L}_{\text{total}} = \lambda_{\text{det}}\mathcal{L}_{\text{det}} + \lambda_{\text{id}}\mathcal{L}_{\text{id}} + \lambda_{\text{vq}}\mathcal{L}_{\text{vq}}\), with weights set to 1.0, 2.0, and 0.1, respectively.

\(\mathcal{L}_{\text{det}}\): Standard Deformable DETR detection loss (classification loss + GIoU loss + L1 regression loss)
\(\mathcal{L}_{\text{id}}\): Identity classification cross-entropy loss (including an "unknown" class), formulating trajectory association as a classification task
\(\mathcal{L}_{\text{vq}}\): Standard VQ loss \(\|\text{sg}[z] - e\|_2^2 + \beta\|z - \text{sg}[e]\|_2^2\), where the first term updates the codewords, and the second term (commitment loss) constrains the encoder output to stay close to the codewords; gradients are backpropagated using the straight-through estimator (STE)

The training employs a U-shaped curriculum strategy: the first 5% of iterations undergo normal training to establish basic tracking capabilities; from 5% to 35%, a dropout of 0.35 is applied to visual embeddings to force the model to rely on geometric and identity cues; after 35%, the visual dropout is reduced to 0.1 to recover appearance details. COCO pre-trained weights are used for initialization, and the AdamW optimizer is employed (lr=\(10^{-4}\), weight decay=\(5\times10^{-4}\)) with a linear warmup in the first epoch.

Key Experimental Results¶

Main Results¶

Comparison on AnimalTrack and BFT (Table 1): HieDG achieves the best HOTA, AssA, and IDF1 on both datasets. Notably, the continuous geometric version HieDG* obtains an IDF1 of only 61.8 on AnimalTrack (largely on par with MOTIP's 61.7, while its AssA drops to 54.2). In contrast, after discretization, the IDF1 and AssA increase to 64.4 and 58.4, respectively—directly validating the core argument that continuous geometry is harmful in low-discriminability scenarios, and discretization is necessary to unleash the value of geometric information.

Method	AnimalTrack HOTA	AnimalTrack AssA	AnimalTrack IDF1	BFT HOTA	BFT AssA	BFT IDF1
SORT	42.5	37.0	49.1	61.2	62.3	77.2
ByteTrack	48.1	49.3	55.5	62.5	64.1	82.3
OC-SORT	-	-	-	66.8	68.7	79.3
MOTR	49.5	47.7	54.1	64.2	64.7	75.2
MOTIP (baseline)	54.1	55.0	61.7	69.2	67.3	80.1
CO-MOT	55.3	55.5	62.1	69.0	67.7	79.0
HieDG* (Continuous Geo.)	54.6	54.2	61.8	69.5	68.8	79.8
HieDG	56.2	58.4	64.4	71.3	72.2	82.5

HieDG also achieves optimal performance on the BuckTales drone scenario (HOTA 52.4, AssA 64.0, IDF1 69.7), significantly outperforming ByteTrack (HOTA 49.8) and MOTIP (HOTA 45.6). HieDG also obtains competitive generalization results on DanceTrack and SportsMOT (DanceTrack HOTA 70.3, SportsMOT HOTA 72.8), indicating that discrete geometric modeling is not limited to animal scenarios.

Ablation Study¶

Geometric components ablation (BFT dataset, with MOTIP as baseline HOTA 69.2):

Configuration	HOTA	AssA	IDF1	Description
MOTIP baseline	69.2	67.3	80.1	Pure appearance, no geometry
+ Position (Pos.)	70.6	71.0	81.6	Position geometry only
+ Scale (Scale)	70.1	68.9	81.3	Scale geometry only
+ Velocity (Vel.)	69.7	68.0	80.9	Velocity geometry only
+ All Three	71.3	72.2	82.5	Maximum complementary gain

Quantization strategy ablation:

Strategy	HOTA	AssA	IDF1	Parameter Increments
Continuous Encoding (No Quantization)	69.5	68.8	79.8	+199.2K
Fixed Bin Quantization (FixVQ)	69.8	69.1	80.2	+49.9K
Single-level VQ	70.5	68.9	81.3	+99.1K
Two-level Residual VQ	71.3	72.2	82.5	+148.3K
Three-level Residual VQ	71.5	72.2	82.7	+197.4K

Key Findings¶

Discretization is a prerequisite for geometric modeling to be effective: The AssA of continuous geometric embedding (HieDG*) on AnimalTrack is even lower than the pure appearance baseline of MOTIP (54.2 vs. 55.0), indicating that unquantized geometric signals act as noise rather than helpers for the attention mechanism. After discretization, the AssA outperforms the baseline by 3.4 percentage points, truly unleashing the value of geometric information.
Position contributes the most, and the three are complementary: Adding position alone improves HOTA by 1.4, while velocity alone only increases it by 0.5; however, combining all three yields a 2.1 improvement, demonstrating that position is the primary signal while scale and velocity provide complementary fine-grained discriminative information.
Two-level VQ offers the best performance-efficiency trade-off: Single-level VQ lacks sufficient representational capacity (70.5 HOTA), while three-level VQ exhibits diminishing marginal returns (71.5 vs. 71.3, an improvement of only 0.2), making the two-level design the optimal balance between performance and efficiency.
Hungarian matching offers no additional gain to the ID classification framework: The MOTIP-style ID classification implicitly completes the global assignment, rendering external Hungarian matching algorithms virtually useless.
t-SNE visualization support: The quantized geometric embeddings exhibit significantly tighter intra-ID clustering than continuous appearance features, and the fused embeddings further widen the margins between different IDs.

Highlights & Insights¶

Clever transfer of the signal processing perspective: Borrowing quantization from signal processing/image compression for geometric modeling in multi-object tracking is clear and intuitive—since perturbations are amplified through attention mechanisms, "flattening" continuous signals into discrete states at the representation level suppresses perturbation propagation at the source. This insight could be applicable to other scenarios that require injecting noise-sensitive signals into attention mechanisms.
Empirical proof of the adverse effects of continuous geometry is warning-worthy: The HieDG* experiment directly proves that "stuffing more information into the model is not always better"—untreated geometric signals can act as poison rather than fuel in attention architectures. This cautions future work to meticulously consider the noise characteristics of auxiliary modalities and the sensitivity of the architecture when introducing auxiliary modalities.
Polar velocity decoupling is a low-cost, highly effective trick: Converting Cartesian velocity to polar coordinates requires only a single-line formula but achieves stable performance gains. It essentially prefaces the task of "decoupling entangled representations"—which neural networks struggle with—into manual feature engineering, easing the learning burden. A similar idea can be transferred to other motion modeling tasks requiring direction and magnitude representation.
The U-shaped curriculum learning strategy is highly reusable: Allowing the model to first master the visual baseline, then gradually reducing visual weights to force the utilization of geometric cues, and finally recovering visual details—this curriculum design of "starting with the backbone, filling in the gaps, and culminating in fusion" is widely applicable when introducing auxiliary modalities.

Limitations & Future Work¶

Weak discriminability in scale embeddings: t-SNE visualizations show clear separation for position and velocity embeddings across different IDs, but the inter-class distance and intra-class compactness of scale embeddings are sub-optimal. The authors suggest exploring area-shape decomposition, log-space encoding, or aspect ratio representations to improve scale modeling.
Simple fusion strategy: The current fusion of geometric and visual features is limited to concatenation + linear projection. Future work could investigate more expressive fusion mechanisms like gated fusion, FiLM modulation, cross-attention, or graph message passing.
ID switches still occur in occlusion and boundary scenarios: Missed detections and identity switches still arise when targets are near image boundaries or partially occluded. Drastic appearance changes in animals (e.g., diving ducks, severe pose changes in deer) can easily cause the system to misclassify the same target as a newly appearing individual.
Lack of long-term trajectory modeling: The current historical time window only covers \([t-T, t-1]\), limiting the ability to recover identity for targets that reappear after prolonged occlusion.

vs. MOTIP (Gao et al., CVPR 2025): MOTIP formulates MOT as an identity classification problem and serves as the baseline for HieDG. HieDG increases discrete geometric branches on top of its pure appearance architecture, demonstrating that geometric information can stably compensate for indistinguishable appearances. Insight: The identity classification framework can naturally absorb multimodal information without altering its core matching logic.
vs. OC-SORT (Cao et al., CVPR 2023): OC-SORT handles irregular motion through motion continuity modeling and direction smoothing, representing heuristic geometric modeling. HieDG encodes similar motion physical intuitions (directional consistency, spatial proximity) into the kernel function design of its KNN velocity estimation, yet remains an end-to-end learnable framework. Insight: Geometric intuitions of heuristic methods can guide feature design within learnable frameworks.
vs. VQ-VAE / Residual Quantization (Lee et al., CVPR 2022): The hierarchical residual codebook design directly draws inspiration from the VQ-VAE line of work, but the application scenario is entirely different—translating from image/audio compression to geometric representation learning in tracking. Insight: Representation discretization techniques from generative models can be transferred to discriminative tasks as robust representation learning tools.

Rating¶

Novelty: ⭐⭐⭐⭐ Introducing quantization from signal processing into the geometric modeling of multi-object tracking is a novel perspective transfer. The core method (residual VQ) itself consists of mature techniques, making the combination clever but not revolutionary.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Five datasets (3 animal + 2 generic), extensive ablations (geometric components, quantization strategies, codebook dimensions, Hungarian matching, polar coordinates), t-SNE visualizations, and a dedicated design for inference velocity estimation, showcasing comprehensive experimental coverage.
Writing Quality: ⭐⭐⭐⭐ The chain of motivation is clear (insufficient appearance \(\rightarrow\) add geometry \(\rightarrow\) continuous geometry is harmful \(\rightarrow\) resolved via discretization). The comparison experiments with HieDG* make the core arguments highly convincing, though some paragraphs are slightly verbose.
Value: ⭐⭐⭐⭐ Provides an effective geometric modeling solution for multi-animal tracking; the empirical proof that continuous geometry hurts has warning value for the community. The concept of discrete geometric representation is transferable to other tracking tasks (pedestrians, vehicles, cell tracking).