PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking¶

Conference: ECCV 2026
arXiv: 2606.30476 ⚠️ Suspected placeholder/future date (2026-06), subject to the original text
Code: https://github.com/xifen523/PS-MOT
Area: Multi-Object Tracking / Weakly-Supervised / Point-Supervision / Object Detection
Keywords: Point-supervised tracking, pseudo-label evolution, wavelet frequency domain, uncertainty modeling, SAM

TL;DR¶

Replacing the per-frame dense box annotations of multi-object tracking with "one point per object" point annotations, this work uses a three-layer design—SAM closed-loop, frequency-domain boundary hallucination, and uncertainty Gaussian loss—to progressively cultivate scale-less point seeds into scale-accurate, identity-consistent instance representations. It achieves 52.3 HOTA on DanceTrack under pure point supervision, saving approximately 64% of annotation costs.

Background & Motivation¶

Multi-object tracking (MOT) has long relied on frame-by-frame, object-by-object dense bounding box annotations. Current mainstream Tracking-by-Detection (e.g., ByteTrack, OC-SORT) and query-based Transformers (e.g., MOTR, MOTIP) achieve high precision on standard benchmarks, but at the cost of requiring massive precise boxes: annotating a box requires precise alignment of the target boundary and camera perspective, while maintaining identity consistency and trajectory continuity across frames, resulting in human labor costs that explode with scale. More troublesome is that in scenarios with severe geometric distortion like embodied robotics or panoramic/fisheye cameras, rigid axis-aligned bounding boxes fail to faithfully describe the true geometric structures, making the "precise box" itself vaguely or even ill-defined. This heavily locks MOT research into a few carefully annotated data domains, making scaling up impossible.

A natural idea is to resort to cheaper supervision signals—points. The paper shows a cost comparison where each point takes about 0.7–0.9 seconds, compared to 7–10 seconds per bounding box. Moreover, as topological centers, points are naturally robust to perspective distortion. However, "point-supervised MOT" (termed PS-MOT) inevitably collides with a precision-ambiguity paradox: a scale-less point is deterministic in location but completely underdetermined in scale and boundary. This spatial ambiguity detonates two specific issues in temporal modeling—spatial leakage (lacking boundary constraints and physical occupancy understanding, supervision signals in crowded scenarios bleed into the background or neighboring targets, diluting discriminative power) and identity drift (scale is implicitly guessed by the model, leading to unstable cross-frame association, frequent trembling, and ID switches). Existing point-supervised works (such as P2BNet, Point2RBox series) almost exclusively estimate scales independently frame-by-frame in static detection, which directly introduces temporal trembling and identity instability when transferred to video.

The Key Insight of this work is: points indeed lack explicit spatial awareness, but the synergy of foundation model priors (SAM) and temporal dynamics can "compensate" for the missing scale representation in a probabilistic space, allowing a pure topological center to progressively evolve into a precise and identity-consistent instance. Core Idea: Deconstruct the point-to-instance process into three stages of coarse-to-fine instance evolution—data, model, and loss levels. At the data level, temporal feedback is used to cultivate points into pseudo-boxes; at the model level, discrete wavelet transform uses point activation to "hallucinate" boundaries in high frequencies; at the loss level, uncertainty Gaussian modeling dynamically adjusts the supervision intensity of each pseudo-label.

Method¶

Overall Architecture¶

PS-Track is a unified framework centered around an "Evolution-Perception-Adaptation" pipeline, progressively approximating dense pixel-level understanding from sparse point supervision. The input is a sparse annotation of one point per target (automatically synthesized from GT box centers in experiments without any manual refinement), and the output is scale-accurate, identity-consistent tracking trajectories. The entire pipeline consists of a three-layer synergy during the training stage:

Data level (TFP, evolution): Instead of treating points as static anchors, it interacts with SAM in a closed loop to convert static point annotations into temporally consistent pseudo-boxes. The key is introducing negative spatial cues to separate adjacent targets, suppressing segmentation fragments via motion-guided box constraints, and assigning a joint quality score to each pseudo-box.
Model level (PEWA, perception): The network's internal representations must also adapt to the sparseness of point supervision. Discrete Wavelet Transform (DWT) is used to decompose features into low-frequency (global semantics) and high-frequency (local boundaries) components, letting the annotated points act as "frequency exciters" to activate high-frequency coefficients only near the targets, thereby "hallucinating" sharp boundaries under sparse signals—and this is entirely bypassed during inference.
Loss level (UGL, adaptation): Acknowledging that pseudo-labels are inherently noisy, regression is converted from deterministic to probabilistic. Pseudo-boxes are modeled as Gaussian observations, using the joint quality score from the data level to derive variance, automatically down-weighting unreliable samples.

A continuous information flow runs through the three levels: the joint quality score produced by TFP is not only used for soft labeling at the data level but also propagates to the loss level to determine the variance (supervision strength) of each sample, threading "data to loss" into a closed loop.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Point Seeds<br/>One point per object"] --> B["Temporal Feedback Prompting TFP<br/>Negative points + motion prior → pseudo-boxes + quality score"]
    B --> C["Point-Excited Wavelet Attention PEWA<br/>Frequency activation of high-freq · hallucinating boundaries"]
    C --> D["Uncertainty Gaussian Learning UGL<br/>Quality score → variance · soft supervision"]
    B -->|Joint quality score S_joint| D
    D --> E["Tracker MOTIP<br/>Inference: Point-free, RGB only"]

Key Designs¶

1. Temporal Feedback Prompting (TFP): Letting SAM Cultivate Static Points into Temporally Consistent Pseudo-Boxes

The performance of point-supervised learning relies heavily on the quality of pseudo-labels. Directly using SAM to generate pseudo-boxes in tracking scenarios encounters two pitfalls: identity merging (a mask incorrectly encircling multiple crowded targets) and 语义碎片 (segmenting only a single limb under occlusion). TFP cures these issues using spatial-temporal constraints. First, negative cues for spatial disambiguation: for target \(i\), center points of its spatial neighbors (trajectories co-existing in the current frame with a center distance smaller than threshold \(\tau_{dist}\)) are sampled as negative prompts \(\mathcal{P}_i^-\), which are fed into SAM alongside the positive point. This acts as a "semantic firewall" for segmentation, forcing the model to reject features of adjacent identities and resolving identity merging. Second, motion priors for temporal regularization: each tracklet maintains its motion state via Kalman filtering. Before segmentation, the state is projected onto the current frame to obtain a motion prior box \(B_{t|t-1}\), which is input into SAM as a box prompt. This narrows the segmentation search space, suppresses outlier regions, and maintains scale consistency of the mask, thereby suppressing fragmentation and reducing drift.

Thirdly, TFP no longer treats all pseudo-labels as ground truth but assigns a joint quality score to each pseudo-box \(B_{pseudo}\) to quantify its reliability:

\[S_{joint} = S_{sam}\cdot(\alpha\cdot\text{IoU}(B_{pseudo}, B_{t|t-1}) + \beta)\]

where \(S_{sam}\) is the visual confidence provided by SAM, the IoU term measures the consistency between visual segmentation and physical motion priors, and \(\alpha,\beta\) control the contribution of temporal consistency and baseline offset. This score is preserved and passed to the loss level for dynamic down-weighting—boxes with both reliable vision and motion receive high scores and weights, while doubtful boxes get low scores and are tolerated. (Note: The paper references [8] and labels the promptable segmentation model here as "SAM 3", i.e., SAM3. The appendix ablation also compares SAM-v1/v2/v3, ⚠️ please refer to the original text for the exact version.)

2. Point-Excited Wavelet Attention (PEWA): Illuminating High Frequencies in the Wavelet Domain with Points to Hallucinate Boundaries from Sparse Signals

The data level converts points to boxes, but the network's internal representation must be capable of inferring "shape" from a single point. This is the position-boundary mismatch: a point tells you where the target is, but a convolutional kernel struggles to infer its shape without explicit boundary supervision; standard CNNs, constrained by local receptive fields and spatial pooling, find it particularly difficult to reconstruct sharp boundaries from sparse points. PEWA draws inspiration from the top-down mechanism of human vision (high-level abstraction guiding low-level detail perception) and treats annotated points as "frequency exciters" during training to "hallucinate" boundaries in the wavelet domain. First, a single-level Haar Wavelet Transform decomposes the feature map into a low-frequency approximation (corresponding to global semantics, preserving identity consistency) and three high-frequency sub-bands LH/HL/HH (corresponding to edge textures). The issue is that without supervision, high frequencies are dominated by background clutter. Thus, PEWA employs masked frequency modulation to denoise and enhance signals: a spatial Gaussian heatmap is generated around the annotated point to represent the potential target range. After downsampling to the wavelet-domain resolution, it passes through a lightweight modulator \(\phi\), a Sigmoid function to obtain a frequency excitation mask \(M_{exc}\), and is then element-wise multiplied with the three high-frequency sub-bands:

\[\hat{X}_{HF}^{k} = X_{HF}^{k}\odot M_{exc}, \quad k\in\{LH, HL, HH\}\]

This step imposes a prior: only boundaries within the neighborhood of annotated points and local semantic regions are valid. Low-frequency components are not modulated to maintain global identity consistency. Finally, Inverse Discrete Wavelet Transform (IDWT) reconstructs the modulated high-frequency and global contexts, which are then added back as residuals to obtain refined features. Compared to traditional upsampling layers, this frequency-domain reconstruction is highly parameter-efficient. The key engineering detail is: PEWA is only inserted after the backbone during training to force the network to learn boundary-sensitive feature responses, and is entirely bypassed during inference—the model only takes RGB frames and does not touch any points, completely preventing label leakage. The appendix also presents an counter-intuitive finding: randomly activating PEWA during training (structural dropout, probability \(p\)) was expected to bridge the "present in training, absent in inference" structural gap, but it actually degraded performance. This is because the inconsistent feature representation (high-frequency boundaries repeatedly reconstructed and revoked) caused feature jitters, interfering with the downstream Transformer in learning stable identity embeddings. A constant activation of \(p{=}1.0\) is optimal (44.2 HOTA / 30.4 AssA).

3. Uncertainty Gaussian Learning (UGL): Treating Pseudo-Boxes as Gaussian Observations and Dynamically Adjusting Supervision Intensity Based on Quality Scores

Pseudo-labels are inherently noisy, and standard MOT objectives (L1, GIoU) implicitly assume that the ground truth follows a Dirac distribution (zero uncertainty). Applying them rigidly to evolved pseudo-boxes forces the network to overfit to SAM segmentation errors and noise from occlusion drift. UGL replaces deterministic regression with probabilistic likelihood maximization: each pseudo-box is modeled as a Gaussian observation centered at the true mean, with the variance \(\sigma^2\) representing observation noise. The core step links variance with the joint quality score from the data level—temporally consistent and visually plausible trajectories should have small variance (high precision), hence \(\sigma_i = 1/(S_{joint,i}+\epsilon)\). Accordingly, the uncertainty-guided regression loss (Gaussian negative log-likelihood) is formulated as:

\[\mathcal{L}_{reg} = \frac{1}{N}\sum_{i=1}^{N}\left(\frac{1}{2\sigma_i^2}\|B_{pred,i} - B_{pseudo,i}\|_2^2 + \frac{1}{2}\log\sigma_i^2\right)\]

This formulation has a clean physical interpretation: the first term is dynamic reweighting—reliable samples (high \(S_{joint}\), small \(\sigma\)) receive a large weight \(\frac{1}{2\sigma^2}\) to force precise localization, while unreliable samples (e.g., occluded targets) have their weights attenuated, allowing the model to tolerate potential annotation errors; the second term \(\frac{1}{2}\log\sigma^2\), since \(\sigma_i\) is deterministically derived from the quality score, acts as a probabilistic calibration to maintain the absolute likelihood scale across batches with varying pseudo-label quality, preventing the objective from degenerating into a purely relative weighted regression. The final loss is combined with classification loss and identity prediction loss (\(\mathcal{L}_{id}\) follows MOTIP, supervised by generated identity labels): \(\mathcal{L}_{total} = \lambda_{reg}\mathcal{L}_{reg} + \lambda_{cls}\mathcal{L}_{cls} + \lambda_{id}\mathcal{L}_{id}\). With this, PS-Track fits the noise distribution of point-supervised data without requiring manual label cleaning.

Loss & Training¶

PS-Track is built on top of MOTIP, using Deformable DETR + ImageNet pre-trained ResNet-50 with 300 object queries. Trained on a single card RTX 5090 with a batch size of 30 video frames per clip, using AdamW for 10 epochs. The base learning rate is \(1\times10^{-4}\) (\(1\times10^{-5}\) for the backbone), decaying at epochs 6 and 9. Point annotations are automatically synthesized from the GT box centers of each dataset (no manual refinement), deliberately preserving noise where "box centers might fall outside the target or onto adjacent targets" to simulate realistic sloppy clicks. Compared to MOTIP, it only adds moderate overhead: training VRAM 23.7 \(\rightarrow\) 24.2 GB, training duration 7.4 \(\rightarrow\) 8.0 h per epoch, parameters 59.1M \(\rightarrow\) 61.9M (including the training-time PEWA branch); during inference, as both SAM and PEWA are bypassed, FLOPs remain unchanged and FPS is almost identical (29.82 \(\rightarrow\) 29.48).

Key Experimental Results¶

Main Results¶

Four benchmarks: DanceTrack, SportsMOT (severe non-linear motion + uniform appearance), JRDB, and EmboTrack (embodied panorama, extreme self-motion, dense crowds). Metrics include HOTA/DetA/AssA + MOTA/IDF1, with OSPA additionally reported for the panoramic dataset. As a purely point-supervised method, the core appeal of PS-Track is not outperforming fully supervised SOTA, but approaching or even outperforming some classic fully supervised methods while using only point annotations.

Dataset	Metrics	PS-Track (Point-supervised)	Baseline	Description
DanceTrack test	HOTA	52.3	CenterTrack 41.8 / FairMOT 39.7 (Fully supervised)	Outperforms classic fully supervised methods by +10.5 / +12.6, falls behind fully supervised SOTA MOTIP (69.6)
DanceTrack test	IDF1	53.4	—	Identity consistency is highly usable under pure point supervision
SportsMOT test	HOTA	45.2	ByteTrack 62.1 (Fully supervised w/o extra)	Establishes a point-supervised baseline, robust to rapid posture deformation
JRDB test	HOTA	20.72	TrackFormer 19.16 / DiffMOT 19.96 (Fully supervised)	Outperforms two fully supervised methods under panoramic dense crowds
EmboTrack QuadTrack	HOTA / IDF1	33.9 / 38.6	ByteTrack 20.7 / OC-SORT 20.8 (Fully supervised)	Leads significantly under embodied self-motion, verifying the TFP negative cue firewall

Cross-paradigm generalizability (DanceTrack val, Tab. 7) is another major selling point: plugging the core module into three mainstream architectures allows fully supervised baselines to operate under pure point supervision—BYTE+PS-Track 42.9 HOTA (only trailing fully supervised BYTE by 4.2), MOTIP+PS-Track 50.3, AR-MOT+PS-Track 36.9. This shows that the framework acts like a "universal catalyst," unlocking point-supervision capabilities for existing MOT paradigms.

Ablation Study¶

On DanceTrack val, with the full 10-epoch schedule, modules are added incrementally (Tab. 6a):

Configuration	HOTA	AssA	Description
① Naive point-to-box baseline	30.3	20.4	No temporal/uncertainty constraints, poor performance
② +TFP	49.0	37.1	+18.7 HOTA, motion priors + negative cues resolve identity merging/fragmentation, biggest contribution
③ +TFP +UGL	49.4	38.0	Adaptively down-weights residual pseudo-label noise
④ Full (+PEWA)	50.3	39.1	PEWA hallucinates boundaries in the frequency domain, providing higher quality embeddings for association

The necessity ablation study (Tab. 5) is highly illustrative: directly coupling the point-supervised detector Point2RBox-v3 with an off-the-shelf matcher (decoupled TBD) leads to absolute failure (10.8 HOTA, MOTA < -28), because the isolated detector cannot resolve scale ambiguity, yielding boxes too corrupted for any matcher to save; whereas training a YOLOX detector with the PS-Track paradigm + BYTE association directly achieves 42.9 HOTA (+32.1), proving that "coupling perception with temporal evolution" is key to making point-supervised tracking work.

Key Findings¶

TFP is the absolute powerhouse: Adding TFP alone boosts HOTA from 30.3 to 49.0 (+18.7), while PEWA and UGL each contribute about 1 point. This aligns with intuition—the most crucial issue in point supervision is pseudo-label quality, and TFP directly addresses the root causes: identity merging and fragmentation.
Exceptionally robust to click noise: Injecting Gaussian noise into training points, HOTA only slightly drops from 44.2 to 44.1 at 16px, remains 42.8 at 32px, and noticeably drops to 39.8 only at 48px. TFP+PEWA anchors inaccurate clicks back to true semantic boundaries via motion priors and frequency hallucination, while UGL down-weights residual errors, yielding robust performance through synergy.
Foundation model quality directly sets the upper bound: Swapping the SAM backbone in TFP yields 20.7/39.0/50.3 HOTA for SAM-v1/v2/v3, respectively. Stronger promptable segmenters produce more reliable pseudo-labels and better tracking; notably, these SAM models are only used offline for pseudo-label generation and do not enter inference.
Insensitive to hyperparameters, smooth convergence: \(\lambda_{reg}\) fluctuates between 1 and 5, peaking at 44.8 (\(\lambda_{reg}{=}3\)); HOTA rises smoothly from 44.2 at epoch 2 to 50.3 at epoch 10 without late-stage degradation, showing that the uncertainty framework effectively resists memorizing weak-supervision noise.

Highlights & Insights¶

Formalizes "point-supervised MOT" as a paradigm and provides a comprehensive data/model/loss three-layer synergistic solution—it is not a minor module adjustment, but a systematic bridging of the gap between sparse supervision and dense perception in three clear, highly transferable steps.
The training-time "frequency exciter" is a clever trick: Points alone contain very sparse information, but treating them as a "biological spotlight" that illuminates target boundaries in the high-frequency wavelet domain during training forces the network to learn boundaries, while being entirely bypassed during inference. This gains performance with zero inference overhead and zero label leakage. This "strong prior during training, retracted during inference" pattern is highly transferable to other weakly supervised scenarios.
Double duty for the joint quality score: The same \(S_{joint}\) is used for soft-labeling at the data level and as the inverse of variance for dynamic down-weighting at the loss level. Running the "data reliability" signal end-to-end from generation to optimization is far more elegant than manually setting arbitrary weights.
Valuable counter-intuitive negative results: Randomly activating PEWA during training (structural dropout) was expected to bridge the training-inference structural gap, but actually degraded performance due to feature jitter. A constant, deterministic structural prior is more conducive to learning stable identity embeddings than deliberately mimicking the inference state.

Limitations & Future Work¶

Failure in extreme occlusion/entangled limbs: Single points might fall on ambiguous boundaries between interacting targets, causing SAM to generate over-segmented or fragmented pseudo-labels (e.g., intensive wrestling, group dancing), which even TFP's negative cues cannot salvage.
PEWA fails under extreme motion blur: High-frequency edge features are physically destroyed, making it difficult to infer the full physical range without explicit scale priors.
Annotation efficiency is a "controlled estimate" rather than end-to-end measurement: Point labels are synthesized from the center of GT boxes, which is cleaner than real human clicks, and actual clicking time was not measured. TFP also introduces extra offline cost for SAM pseudo-label generation. The claimed ~64% saving should be understood as a "controlled estimate of reduced bounding-box annotation workload." The authors also emphasize that MOT still requires temporal identity linking, and point supervision does not eliminate this burden.
Future directions: Multimodal prompts (points + coarse language descriptions) to resolve semantic ambiguity; temporally sparse annotations (e.g., providing a point every 5 frames) + unsupervised motion propagation to further compress annotations; and translating this topological center concept to 3D MOT (LiDAR/BEV) to unlock autonomous driving-grade scaling.

vs P2BNet / Point2RBox series (static point-supervised detection): These methods use MIL or coarse-to-fine regression to hallucinate boxes from points, but they estimate scales frame-by-frame independently, ignoring temporal continuity. Applying them to video introduces jitters and identity instability. This paper reformulates "point to mask" as a trajectory-aware process and enforces spatial-temporal consistency via motion priors, which is a necessary evolution for point supervision to progress from detection to tracking. The ablation showing the catastrophic failure of Point2RBox-v3 directly connected to a matcher empirically validates this.
vs MOTIP (fully-supervised query-based SOTA): This work is built directly on MOTIP, transforming it from fully-supervised to point-supervised. MOTIP achieves 69.6 HOTA on DanceTrack, while PS-Track achieves 52.3—a gap exists, but in exchange, annotation costs are reduced by orders of magnitude, and it is more practical in scenarios where "boxes themselves are hard to define," such as JRDB/EmboTrack.
vs Unsupervised/Semi-supervised/Weakly-supervised tracking: Unsupervised/semi-supervised methods are often plagued by occlusions or highly dependent on the quality of initial seeds; weakly-supervised methods still rely on box priors to estimate scale. This work completely eliminates dense box annotations, shifting the focus from absolute spatial precision to temporal identity consistency.
vs WaveFormer (frequency-domain visual modeling): WaveFormer performs general frequency decoupling, whereas this work specifically utilizes DWT to isolate high frequencies and uses points as exciters to reconstruct boundaries, focusing primarily on "hallucinating boundaries from sparse points."

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formally proposes and systematically solves the new PS-MOT paradigm. The three-layer design is highly clever; using points as frequency exciters is particularly novel.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Four benchmarks + cross-three paradigms + rich ablation/noise/convergence/SAM variant analyses are very solid; points are deducted because point labels are synthesized from box centers and real human click cost studies were not performed.
Writing Quality: ⭐⭐⭐⭐☆ The logical narrative (Evolution-Perception-Adaptation) is clear, and the motivations are concrete; minor terminology (like labeling SAM versions) is slightly confusing.
Value: ⭐⭐⭐⭐⭐ Frees MOT from the scaling bottleneck of dense box annotations, which is especially beneficial for environments where boxes are hard to define like panoramic/embodied scenarios. Highly promising practical potential.