PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking¶

Conference: ECCV 2026
arXiv: 2606.30476 ⚠️ Suspected placeholder/future date (2026-06), subject to the original text
Code: https://github.com/xifen523/PS-MOT
Area: Multi-Object Tracking / Weakly Supervised / Point Supervision / Object Detection
Keywords: Point-supervised tracking, pseudo-label evolution, wavelet frequency domain, uncertainty modeling, SAM

TL;DR¶

This work replaces dense per-frame bounding box annotations in multi-object tracking with "one point per object" point annotations. Through a three-layer design comprising SAM-in-the-loop, frequency-domain boundary hallucination, and uncertainty Gaussian loss, scale-less point seeds are progressively "cultivated" into scale-accurate, identity-continuous instance representations. Under pure point supervision, it achieves 52.3 HOTA on DanceTrack, reducing annotation costs by approximately 64%.

Background & Motivation¶

Multi-object tracking (MOT) has long relied on dense, per-frame, per-object bounding box annotations. Current dominant tracking-by-detection methods (e.g., ByteTrack, OC-SORT) and query-based Transformers (e.g., MOTR, MOTIP) achieve impressive accuracy on standard benchmarks, but at the cost of requiring massive quantities of precise bounding boxes. Annotating a bounding box requires precise alignment of object boundaries with camera perspectives, as well as maintaining identity consistency and trajectory continuity across frames, resulting in human labor costs that explode with scale. Crucially, in scenes with severe geometric distortion such as embodied robotics or panoramic/fisheye views, rigid axis-aligned boxes cannot faithfully describe the true geometric structures, making the concept of "precise boxes" itself ambiguous or ill-posed. This locks MOT research into a few meticulously annotated data domains, hindering scalability.

A natural alternative is to leverage cheaper supervisory signals—points. The paper notes a cost comparison of approximately 0.7–0.9 seconds per point versus 7–10 seconds per box, and points, acting as topological centers, naturally resist perspective distortion. However, "point-supervised MOT" (termed PS-MOT in this paper) inevitably encounters a precision-ambiguity paradox: while a scale-less point is positionally deterministic, its scale and boundary are completely underdetermined. This spatial ambiguity triggers two specific issues in temporal modeling: spatial leakage (the model lacks boundary constraints and understanding of physical occupancy, causing supervisory signals to leak into the background or neighboring targets in crowded scenes, which dilutes discriminative power) and identity drift (scale is implicitly guessed by the model, leading to unstable cross-frame association, frequent jitters, and ID switches). Existing point-supervised works (e.g., P2BNet, Point2RBox series) almost exclusively estimate scales frame-by-frame independently in static detection, which directly introduces temporal jitter and identity instability when applied to video.

Key Insight: While points lack explicit spatial awareness, the synergy of foundation model priors (SAM) and temporal dynamics can reconstruct the missing scale representation in a probabilistic space, progressively evolving a pure topological center into a precise and identity-consistent instance. Core Idea: The process of mapping points to instances is decomposed into data, model, and loss levels for coarse-to-fine instance evolution. In the data layer, temporal feedback evolves point annotations into pseudo-boxes; in the model layer, point activation in the wavelet frequency domain "hallucinates" boundaries from high frequencies; in the loss layer, uncertainty Gaussian modeling dynamically adjusts the supervision intensity of each pseudo-label.

Method¶

Overall Architecture¶

PS-Track is a unified framework centered around an "Evolution-Perception-Adaptation" pipeline that progressively bridges sparse point supervision to dense pixel-level understanding. The input is sparse point annotations of one point per object (automatically synthesized from GT box centers in experiments, without manual refinement), and the output is scale-accurate, identity-continuous tracking trajectories. The entire pipeline consists of a three-layer synergy during the training phase:

Data layer (TFP, Evolution): Instead of treating points as static anchors, it uses closed-loop interaction with SAM to transform static point annotations into temporally consistent pseudo-boxes. Key aspects include introducing negative spatial cues to separate adjacent targets, suppressing segmentation fragments using motion-induced box constraints, and assigning a joint quality score to each pseudo-box.
Model layer (PEWA, Perception): The inner representation of the network is adapted to the sparseness of point supervision. Discrete Wavelet Transform (DWT) splits features into low frequencies (global semantics) and high frequencies (local boundaries). Annotation points act as "frequency exciters" that selectively activate high-frequency coefficients near target regions, thereby "hallucinating" sharp boundaries under sparse signals—and this module is completely bypassed during inference.
Loss layer (UGL, Adaptation): Acknowledging that pseudo-labels are inherently noisy, regression is formulated probabilistically rather than deterministically. Pseudo-boxes are modeled as Gaussian observations, with variances inferred from the joint quality scores passed down from the data layer, automatically down-weighting unreliable samples.

An information flow runs through these three layers: the joint quality score produced by TFP is not only used for soft labeling at the data layer but also passed to the loss layer to determine the variance (supervision strength) of each sample, creating a closed-loop "data-to-loss" coupling.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Point Seeds<br/>One point per object"] --> B["Temporal Feedback Prompting TFP<br/>Negative points + motion prior → pseudo-boxes + quality score"]
    B --> C["Point-Excited Wavelet Attention PEWA<br/>Frequency activation of high frequencies · boundary hallucination"]
    C --> D["Uncertainty Gaussian Learning UGL<br/>Quality score → variance · soft supervision"]
    B -->|Joint quality score S_joint| D
    D --> E["Tracker MOTIP<br/>Point-free at inference, RGB only"]

Key Designs¶

1. Temporal Feedback Prompting (TFP): Leveraging SAM to Evolve Static Points into Temporally Consistent Pseudo-boxes

How well point supervision learns depends heavily on the quality of pseudo-labels. Directly running SAM in tracking scenarios faces two issues: identity merging (one mask incorrectly enclosing multiple crowded targets) and semantic fragmentation (segmenting only a discriminative localized part, such as a single limb, under occlusion). TFP addresses these via spatial and temporal constraints. First, negative cues for spatial disambiguation: for object \(i\), its spatial neighbors (trajectories co-existing in the current frame with center distances less than a threshold \(\tau_{dist}\)) are sampled as negative prompts \(\mathcal{P}_i^-\), which are fed into SAM alongside the positive point. This acts as a semantic "firewall" for segmentation, forcing the model to reject features belonging to neighboring identities, directly resolving identity merging. Second, motion priors for temporal regularization: each tracklet maintains its motion state via a Kalman filter. Before segmentation, the state is projected onto the current frame to obtain a motion prior box \(B_{t|t-1}\), which is input into SAM as a box prompt. This narrows the segmentation search space, suppresses outlier regions, and enforces scale consistency across frames, thereby mitigating fragmentation and reducing drift.

Third, TFP stops treating all pseudo-labels as ground truth, and instead assigns a joint quality score to each pseudo-box \(B_{pseudo}\) to quantify its reliability:

\[S_{joint} = S_{sam}\cdot(\alpha\cdot\text{IoU}(B_{pseudo}, B_{t|t-1}) + \beta)\]

where \(S_{sam}\) is the inherent visual confidence from SAM, and the IoU term measures the consistency between visual segmentation and physical motion priors. \(\alpha, \beta\) control the contribution of temporal consistency and the baseline offset, respectively. This score is retained and passed to the loss layer for dynamic down-weighting—boxes with consistent visual and motion properties receive high scores and larger weights, while questionable boxes receive low scores and are tolerated. (Note: The paper cites [8] referencing this promptable segmentation model as "SAM 3", i.e., SAM3, and the appendix ablation also compares SAM-v1/v2/v3, ⚠️ subject to the original text for specific versions.)

2. Point-Excited Wavelet Attention (PEWA): Activating High Frequencies via Points in the Wavelet Domain to Hallucinate Boundaries from Sparse Signals

The data layer transforms points into boxes, but the network's internal representation must also be capable of inferring "shape" from a single point. This points to the position-boundary mismatch: while points indicate where an object is, convolution kernels struggle to infer its shape without explicit boundary supervision. Standard CNNs, limited by local receptive fields and spatial pooling, find it challenging to reconstruct sharp boundaries from sparse points. PEWA draws inspiration from the top-down mechanism of human vision (high-level abstraction guiding low-level detail perception) and treats annotation points as training-time "frequency exciters" to hallucinate boundaries in the wavelet domain. First, a single-level Haar Wavelet Transform decomposes the feature map into a low-frequency approximation (corresponding to global semantics, preserving identity consistency) and three high-frequency subbands, LH/HL/HH (corresponding to edge textures). Unsupervised high frequencies are usually dominated by background noise, so PEWA performs masked frequency modulation to denoise and enhance signals: a spatial Gaussian heatmap centered on the annotation point is generated to represent the potential target range. After downsampling to the wavelet-domain resolution, it passes through a lightweight modulator \(\phi\) and a Sigmoid activation to produce a frequency exciting mask \(M_{exc}\), which is then element-wise multiplied with the three high-frequency subbands:

\[\hat{X}_{HF}^{k} = X_{HF}^{k}\odot M_{exc}, \quad k\in\{LH, HL, HH\}\]

This step enforces a prior: only boundaries within the neighborhood of the annotation point and local semantic regions are valid. The low-frequency component is left unmodulated to preserve global identity consistency. Finally, an Inverse Discrete Wavelet Transform (IDWT) reconstructs the modulated high frequencies with global context, which is added back to the residual to yield refined features. Compared to traditional upsampling layers, this frequency-domain reconstruction requires fewer parameters. Crucially, PEWA is only inserted after the backbone during training to force the network to learn boundary-sensitive feature responses; it is completely bypassed during inference, where the model only processes RGB frames without using any points, eliminating label leakage. The appendix also shares a counter-intuitive finding: randomly activating PEWA during training (structural dropout with probability \(p\)) was expected to bridge the "train-with, infer-without" structural gap, but actually caused a performance drop. This occurs because inconsistent feature representations (high-frequency boundaries being repeatedly reconstructed and discarded) cause feature jitter, disrupting the downstream Transformer's ability to learn stable identity embeddings. A constant activation of \(p{=}1.0\) proved optimal (44.2 HOTA / 30.4 AssA).

3. Uncertainty Gaussian Learning (UGL): Treating Pseudo-boxes as Gaussian Observations with Dynamically Adjusted Supervision Strength via Quality Scores

Pseudo-labels are inherently noisy, whereas standard MOT objectives (\(L1\), \(GIoU\)) implicitly assume that the ground truth follows a Dirac distribution (zero uncertainty). Applying this rigidly to evolved pseudo-boxes forces the network to overfit to SAM segmentation errors and noise from occlusion drift. UGL replaces deterministic regression with probabilistic likelihood maximization by modeling each pseudo-box as a Gaussian observation centered on the true mean, with variance \(\sigma^2\) representing observation noise. A key step links the variance directly to the joint quality score from the data layer: temporally consistent and visually reliable trajectories should have narrow variances (high accuracy), thus \(\sigma_i = 1/(S_{joint,i}+\epsilon)\). This yields the uncertainty-guided regression loss (Gaussian negative log-likelihood):

\[\mathcal{L}_{reg} = \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{2\sigma_i^2}\|B_{pred,i} - B_{pseudo,i}\|_2^2 + \frac{1}{2}\log\sigma_i^2\right)\]

This formula has a clean physical interpretation: the first term acts as dynamic reweighting—reliable samples (high \(S_{joint}\), low \(\sigma\)) receive larger weights \(\frac{1}{2\sigma^2}\) to force precise localization, while unreliable samples (e.g., occluded targets) receive decayed weights, allowing the model to tolerate potential annotation errors; the second term \(\frac{1}{2}\log\sigma^2\), with \(\sigma_i\) deterministically derived from the quality score, serves for probabilistic calibration. This maintains absolute likelihood scale across batches with varying pseudo-label quality, preventing the objective from collapsing into a purely relative weighted regression. The total objective is combined with classification and identity prediction losses (where \(\mathcal{L}_{id}\) follows MOTIP, supervised by generated identity labels): \(\mathcal{L}_{total} = \lambda_{reg}\mathcal{L}_{reg} + \lambda_{cls}\mathcal{L}_{cls} + \lambda_{id}\mathcal{L}_{id}\). Through this, PS-Track adapts to the noise distribution of point-supervised data without manual label cleansing.

Loss & Training¶

PS-Track is built on top of MOTIP, utilizing Deformable DETR with an ImageNet-pretrained ResNet-50 and 300 object queries. It is trained on a single RTX 5090 GPU with a batch size of 30 video frames per clip, using the AdamW optimizer for 10 epochs. The base learning rate is set to \(1\times10^{-4}\) (\(1\times10^{-5}\) for the backbone) and decayed at epochs 6 and 9. Point annotations are automatically synthesized from the centers of the ground-truth boxes in each dataset (without manual refinement), intentionally retaining noise where the "box center might fall outside the target or onto an adjacent target" to simulate realistic sloppy clicks. Compared to MOTIP, only moderate overhead is introduced: training memory increases from 23.7 to 24.2 GB, training time per epoch goes from 7.4 to 8.0 h, and parameters increase from 59.1M to 61.9M (including the training-time PEWA branch). During inference, since both SAM and PEWA are bypassed, the FLOPs remain unchanged, and the FPS is nearly identical (29.82 vs. 29.48).

Key Experimental Results¶

Main Results¶

Evaluation is conducted on four benchmarks: DanceTrack, SportsMOT (severe non-linear motion + uniform appearance), JRDB, and EmboTrack (embodied panoramic, extreme egomotion, dense crowds). Metrics reported include HOTA/DetA/AssA, MOTA/IDF1, and OSPA for panoramic datasets. As a purely point-supervised method, the core value of PS-Track is not necessarily to outperform fully supervised SOTA, but to approach or even outperform some classic fully supervised methods under pure point supervision.

Dataset	Metrics	PS-Track (Point-supervised)	Baseline	Description
DanceTrack test	HOTA	52.3	CenterTrack 41.8 / FairMOT 39.7 (fully supervised)	Outperforms classic fully supervised by +10.5 / +12.6; trails fully supervised SOTA MOTIP (69.6)
DanceTrack test	IDF1	53.4	—	Identity consistency is viable under pure point supervision
SportsMOT test	HOTA	45.2	ByteTrack 62.1 (fully supervised w/o extra)	Establishes the point-supervised baseline; robust to rapid pose deformation
JRDB test	HOTA	20.72	TrackFormer 19.16 / DiffMOT 19.96 (fully supervised)	Outperforms two fully supervised methods in panoramic dense crowds
EmboTrack QuadTrack	HOTA / IDF1	33.9 / 38.6	ByteTrack 20.7 / OC-SORT 20.8 (fully supervised)	Significantly leads under embodied egomotion, validating the TFP negative-cue firewall

Cross-paradigm generalizability (DanceTrack val, Tab. 7) is another major selling point: when inserting the core modules into three major architectures, the fully supervised baselines can operate under pure point supervision—yielding 42.9 HOTA for BYTE+PS-Track (only lagging behind fully supervised BYTE by 4.2), 50.3 for MOTIP+PS-Track, and 36.9 for AR-MOT+PS-Track. This demonstrates that the framework acts like a "universal catalyst," unlocking point supervision capabilities for existing MOT paradigms.

Ablation Study¶

DanceTrack val, under a full 10-epoch schedule with step-by-step module additions (Tab. 6a):

Configuration	HOTA	AssA	Description
① Naive point-to-box baseline	30.3	20.4	No temporal/uncertainty constraints; poor performance
② +TFP	49.0	37.1	+18.7 HOTA; motion priors + negative cues resolve identity merging/fragmentation; largest contributor
③ +TFP +UGL	49.4	38.0	Adaptively downweights residual pseudo-label noise
④ Full (+PEWA)	50.3	39.1	PEWA hallucinates boundaries in the frequency domain, providing higher-quality embeddings for association

Another necessity ablation (Tab. 5) is highly illustrative: directly coupling a point-supervised detector Point2RBox-v3 with an off-the-shelf tracker (decoupled TBD) leads to a catastrophic failure (10.8 HOTA, MOTA < -28), because isolated detectors cannot resolve scale ambiguity, yielding boxes so poor that no association algorithm can salvage them. Conversely, training a YOLOX detector with the PS-Track paradigm + BYTE association directly achieves 42.9 HOTA (+32.1), proving that "unification of perception and temporal evolution" is key to making point-supervised tracking work.

Key Findings¶

TFP is the primary contributor: Introducing TFP alone boosts HOTA from 30.3 to 49.0 (+18.7), whereas PEWA and UGL each provide an additional ~1 point improvement. This aligns with intuition—the most critical issue in point supervision is pseudo-label quality, and TFP directly addresses its root causes: identity merging and semantic fragmentation.
Exceptionally robust to click noise: Injecting Gaussian perturbation to training points only slightly reduces HOTA from 44.2 to 44.1 at 16px, maintains 42.8 at 32px, and noticeabily drops to 39.8 only at 48px. The synergy of TFP+PEWA (using motion prior and frequency-domain hallucination to anchor imprecise clicks back to realistic semantic boundaries) and UGL (down-weighting residual errors) underpins this robustness.
The quality of foundation models directly dictates the upper bound: Swapping the SAM backbone in TFP yields 20.7, 39.0, and 50.3 HOTA for SAM-v1, SAM-v2, and SAM-v3, respectively. Stronger promptable segmentors produce more reliable pseudo-labels, leading to better tracking, and these SAM models are only used for offline pseudo-label generation, not during inference.
Insensitive to hyperparameters and converges smoothly: Varying \(\lambda_{reg}\) between 1 and 5 yields a peak HOTA of 44.8 (at \(\lambda_{reg}{=}3\)). HOTA rises smoothly from 44.2 at epoch 2 to 50.3 at epoch 10 without late-stage degradation, showing that the uncertainty framework effectively resists memorizing weak supervision noise.

Highlights & Insights¶

Formulating "point-supervised MOT" as a formal paradigm and providing a complete, systematic solution across the three layers of data/model/loss. Rather than minor block modifications, this systematically bridges the gap between sparse supervision and dense perception in three clear, transferable steps.
The training-time "frequency exciter" is an elegant design: Points contain sparse information, but treating them as "biological spotlights" to light up target boundaries in the wavelet high-frequency domain forces the network to learn boundaries during training, while bypassing the module completely during inference. This achieves performance gains with zero inference overhead and zero label leakage. This "strong prior during training, removed during inference" pattern is highly transferable to other weakly supervised scenarios.
Twofold utility of the joint quality score: The exact same \(S_{joint}\) is used for soft labeling at the data layer and serves as the inverse of variance for dynamic down-weighting at the loss layer. Propagating the "data reliability" signal end-to-end from generation to optimization is far more elegant than manually tuning heuristic weights.
Counter-intuitive negative results offer value: Randomly activating PEWA during training (structural dropout) was expected to bridge the train-infer structural discrepancy, but actually dropped performance due to feature jitter. Constant deterministic structural priors are more beneficial for learning stable identity embeddings than trying to mimic the inference-stage structures.

Limitations & Future Work¶

Failures under extreme occlusion/tangled limbs: Single points might fall on ambiguous boundaries between targets, causing SAM to generate over-segmented or fragmented pseudo-labels (e.g., dense wrestling, group dance formations), which TFP's negative cues fail to resolve.
PEWA fails under extreme motion blur: High-frequency edge features are physically destroyed, making it difficult to infer complete physical extents without explicit scale priors.
Annotation efficiency is a "controlled estimation" rather than end-to-end measurement: Point labels are synthesized from GT box centers, which are cleaner than real human clicks, and actual clicking times were not measured. Furthermore, TFP introduces additional offline SAM pseudo-label generation costs. The claimed ~64% reduction should be interpreted as a controlled estimate of reduced box annotation workload. The authors emphasize that MOT still requires temporal identity linking, which point supervision does not completely eliminate.
Future directions: Multimodal prompts (points + rough language descriptions) to resolve semantic ambiguities; temporally sparse annotations (e.g., providing a point every 5 frames) + unsupervised motion propagation to further compress annotation workloads; and extending this topological center concept to 3D MOT (LiDAR/BEV) to unlock scalability for autonomous driving.

vs. P2BNet / Point2RBox series (Static point-supervised detection): These methods use MIL or coarse-to-fine regression to hallucinate boxes from points, but estimate scales frame-by-frame independently, ignoring temporal continuity. When applied to video, this introduces jitter and identity instability. The proposed method reformulates "point to mask" as a trajectory-aware process and uses motion priors to force spatiotemporal consistency, marking a necessary evolution of point supervision from detection to tracking. The catastrophic failure of Point2RBox-v3 when directly coupled with a tracker in the ablations supports this.
vs. MOTIP (Fully supervised query-based SOTA): This work is directly built on top of MOTIP, adapting it from fully supervised to point-supervised. While MOTIP achieves 69.6 HOTA on DanceTrack and PS-Track gets 52.3, the gap is traded for a massive reduction in annotation costs. Furthermore, in scenarios like JRDB/EmboTrack where "boxes are hard to define," this point-supervised approach is actually more practical.
vs. Unsupervised / Semi-supervised / Weakly-supervised tracking: Unsupervised/semi-supervised methods often suffer from occlusions or heavily rely on initial seed quality, while weakly-supervised tracking still relies on box priors for scale estimation. PS-Track completely bypasses dense box annotations, shifting focus from absolute spatial precision to temporal identity consistency.
vs. WaveFormer (Frequency-domain visual modeling): WaveFormer performs general frequency decoupling, whereas this work specifically leverages DWT to isolate high frequencies and uses points as exciters to reconstruct boundaries, focusing strictly on "hallucinating boundaries from sparse points."

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates and systematically solves the new PS-MOT paradigm. The three-layer design is highly clever, particularly treating points as frequency exciters.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Solid evaluation across four benchmarks, three tracking paradigms, and rich ablations analyzing noise, convergence, and SAM variants. Slight deduction for synthesizing point labels from box centers without investigating actual human clicking costs.
Writing Quality: ⭐⭐⭐⭐☆ Clear logical flow (evolution-perception-adaptation) and concrete motivations. Individual terms (such as citing SAM versions) could be slightly confusing.
Value: ⭐⭐⭐⭐⭐ Liberates MOT from the scaling bottleneck of dense box annotations. It is particularly useful for panorama and embodied settings where boxes are hard to define, showing high practical potential.