PLOT: Pseudo-Labeling via Object Tracking for Monocular 3D Object Detection¶

Conference: ECCV 2026
arXiv: 2507.02393
Project Page: https://plot-eccv.github.io
Code: To be confirmed
Area: Autonomous Driving / 3D Vision
Keywords: Monocular 3D Object Detection, Pseudo-Labeling, Object Tracking, Dense Point Tracking, Shape Fusion

TL;DR¶

PLOT proposes a 3D pseudo-labeling framework that requires no auxiliary sensors or model retraining. By establishing cross-frame point-level correspondences via dense point tracking, decoupling camera ego-motion and object motion estimation from background/object trajectories, and introducing a global object memory to maintain long-term identity consistency, the framework aligns and fuses multi-view partial observations into complete object pseudo-point clouds. It achieves performance comparable to fully supervised methods on benchmarks like KITTI, KITTI-360, and Waymo, and generalizes well to non-driving scenarios.

Background & Motivation¶

Monocular 3D object detection (M3OD) aims to recover the 3D geometry, pose, and scale of objects from a single RGB image. It holds significant value in fields like autonomous driving, robotics, and surveillance due to its low sensor cost. However, this problem is inherently ill-posed—a single image suffers from depth-scale ambiguity and lacks direct geometric supervision. Consequently, existing methods rely almost entirely on carefully annotated datasets collected in sensor-rich environments (e.g., KITTI, Waymo), where 3D ground truth depends on LiDAR point clouds or multi-view calibration. When models are deployed in unconstrained scenarios—such as handheld cameras, surveillance perspectives, and non-driving scenes—their zero-shot performance undergoes a drastic decline. Unknown camera motion, diverse perspectives, and frequent occlusions cause single-frame assumptions to fail entirely.

To alleviate the scarcity of 3D annotations, pseudo-labeling and weakly-supervised methods have been proposed to generate 3D supervision without LiDAR or multi-view data. However, almost all existing pipelines operate on single-frame images. Single-frame observations only provide partial and noisy geometric information (e.g., only seeing one side of an object), leaving occlusion and scale ambiguity fundamentally unresolved. More importantly, many methods depend on known sensor poses or carefully designed geometric priors (such as vehicle dimension templates), which severely restricts cross-domain scalability. If the camera pose in a new scenario is unknown, these methods become completely unusable.

This work encounters the insight that monocular videos naturally encode rich temporal geometric cues: viewpoint changes of objects between adjacent frames offer multi-view constraints, and the relative motion between the camera and objects provides additional signals for 3D attribute estimation. However, leveraging videos effectively introduces two core challenges. First, maintaining long-term object identity consistency under unknown poses (as occlusion and detection failures lead to track fragmentation and identity switches). Second, fusing partial observations from multiple frames into a complete object geometry (since single-frame observations are always partial and noisy). PLOT addresses these by decoupling object tracking from background tracking, recovering relative motion using dense point correspondences, and maintaining identity consistency with a global object memory, ultimately aligning and fusing multi-frame sparse observations into a complete object shape through simple optimization. Core Idea: Utilize dense point tracking to simultaneously establish cross-frame point-level correspondences for both objects and background, decoupling and estimating camera ego-motion and object motion. A global object memory is introduced to repair detection/tracking failures at the recognition level, and ultimately align and fuse multi-view partial observations into complete pseudo-point clouds—all without requiring auxiliary sensors or model retraining to generate high-quality 3D annotations from pure monocular videos.

Method¶

Overall Architecture¶

PLOT is a pure post-processing pipeline that does not rely on auxiliary sensors or model training. It takes monocular video frames as input and outputs 3D pseudo-labels (position, dimension, orientation) for each frame. The overall pipeline is divided into four stages: first, establishing cross-frame point-level correspondences via dense point tracking (temporal anchoring); second, estimating camera ego-motion via background trajectories and object motion via object trajectories; third, introducing Global Object Memory (GOM) to maintain object identity consistency over long time spans, repairing track fragmentation caused by detection failures or occlusions; and finally, aligning and fusing multi-frame partial observations of the same object in a unified coordinate system to generate a complete pseudo-point cloud, which is then projected back to each frame to extract consistent 3D attributes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Monocular Video Frames<br/>It, t ∈ T"] --> B["Temporal Anchoring<br/>GSAM Detection + Dense Point Tracking<br/>Hungarian Matching: Predicted ↔ Tracked Mask"]
    B --> C["Motion Estimation<br/>Background Points → Procrustes → Camera Ego-Motion<br/>Object Points → Procrustes → Object Motion"]
    C --> D["Global Object Memory (GOM)<br/>Persistent Entries + Discard/Update/Merge Rules"]
    D --> E["Trajectory-Guided Shape Fusion<br/>Align Multi-frame Pseudo-Point Clouds to Ref Frame + Aggregation"]
    E --> F["Output: 3D Pseudo-Labels<br/>Per-frame Position/Dimension/Orientation"]

Key Designs¶

1. Temporal Anchoring: Dual-Track Masks and Dense Point Tracking to Establish Cross-Frame Correspondences

Establishing consistent frame-to-frame object correspondences in videos with unknown poses is the foundation for all subsequent geometric reasoning. The key insight of PLOT is to maintain two complementary mask tracks: predicted masks generated by the open-vocabulary detector GSAM predicting object masks independently per frame, capturing actual shape variations over time; and tracked masks propagated from a mask in one frame to other frames using the dense point tracker AllTracker, preserving point-level correspondences required for motion estimation and shape alignment. Specifically, for the \(k\)-th object mask \(M_k^t\) at frame \(t\), its projection in other frames \(M_k^{\acute{t} \leftarrow t}\) is obtained via dense point tracking. Since detectors may miss detections and trackers may drift, predicted and tracked masks may not perfectly overlap. PLOT performs Hungarian matching to maximize IoU between them to establish final correspondences. Simultaneously, background regions outside the object masks are sampled and tracked via point tracking to obtain background point correspondences, which are dedicated to camera ego-motion estimation. Running both tracks in parallel and sharing the same set of point correspondences during matching is a key prerequisite for the framework's efficiency.

2. Decoupling Motion Recovery from Point Trajectories: Joint Estimation of Camera Ego-Motion and Object Motion

With cross-frame correspondences established, the core challenge is to align all observations into a unified coordinate system. PLOT achieves this by estimating camera motion and object motion separately. For background points, they are lifted to 3D (with the assistance of the monocular depth estimator UniDepth) and then aligned via Procrustes alignment to solve for the camera rotation \(\mathbf{R}_c^t\) and translation \(\mathbf{t}_c^t\) from frame \(t\) to reference frame \(r\):

\[\arg\min_{s_c^t,\mathbf{R}_c^t,\mathbf{t}_c^t} \sum_{i \in M_{bg}^t} \mathcal{M}\mathcal{V} \left\| \mathbf{p}_i^r - (s_c^t \mathbf{R}_c^t \mathbf{p}_i^{r\leftarrow t} + \mathbf{t}_c^t) \right\|^2\]

Where \(\mathcal{M}\) is a binary mask filtering out unreliable depth (e.g., >50m), \(\mathcal{V}\) indicates whether a point is visible in both frames, and \(s_c^t\) is an optional scale factor (activated only when depth estimation has significant temporal drift, otherwise fixed to 1). For object points, a similar Procrustes alignment computes the object-level relative motion. After transforming the object trajectories into the world coordinate system via the estimated camera motion, object motion can be determined: if the displacement between two frames exceeds a threshold, the object is treated as dynamic, and its orientation is given by the motion direction angle; otherwise, it is treated as static, and its orientation is determined by the principal direction obtained via PCA on the fused complete pseudo-point cloud. This complementary "dynamic-from-trajectory, static-from-geometry" strategy keeps orientation estimation reliable across both scenarios.

3. Global Object Memory (GOM): Persistent Identity Consistency Maintenance

Even with point tracking and Hungarian matching, occlusions, missed detections, and camera motion in real-world videos still cause frequent identity switches and track fragmentation—an object reappearing after being occluded for a few frames might be assigned a new ID, or multiple neighboring objects might be merged into a single track, which seriously pollutes subsequent shape fusion. To address this, PLOT designs a lightweight Global Object Memory module. GOM maintains a list of persistent object entries, each recording the complete observation history of a unique object across the timeline. New detection results in each frame are matched to existing memory entries based on spatial overlap with tracked masks. GOM performs three types of checks at each timestep: ① whether the point tracking of the object is continuous (tracks shorter than 5 frames are discarded); ② whether the category and geometric attributes of the object are temporally consistent (inconsistent entries represent noisy detections and are discarded); ③ whether multiple memory entries actually correspond to the same object (and merge them if so). High-confidence detections that do not match any existing entries temporarily create new entries. This rule-based mechanism introduces no learnable parameters and effectively repairs detector misses (by continuing from memory) and identity switches (by merging broken tracks) purely by verifying the continuity of point tracking and spatio-temporal attribute consistency.

4. Trajectory-Guided Shape Fusion: Multi-frame Pseudo-Point Cloud Alignment and Aggregation

The fatal flaw of single-frame observations is that they only capture partial object surfaces—when the side or rear of a vehicle is occluded, center offset and orientation estimation will drift. The core compensation strategy in PLOT is to align and fuse multi-frame observations of the same object. Specifically, a reference frame that minimizes inter-frame registration error is first selected for each object. Then, the 3D points of this object in all other frames are transformed into the coordinate system of the reference frame \(r\) using the previously estimated relative poses to merge them:

\[\hat{\mathbf{P}}_k = \bigcup_{t \in \mathcal{T}} s^{t \rightarrow r} \mathbf{R}^{t \rightarrow r} \mathbf{P}_k^t + \mathbf{t}^{t \rightarrow r}\]

Where \(s^{t \rightarrow r}\), \(\mathbf{R}^{t \rightarrow r}\), and \(\mathbf{t}^{t \rightarrow r}\) represent the scale, rotation, and translation from frame \(t\) to reference frame \(r\). As the camera or object moves, different frames capture different sides of the object, and the fused point cloud \(\hat{\mathbf{P}}_k\) continuously fills in the missing surfaces. Next, PCA is performed on \(\hat{\mathbf{P}}_k\) to extract the principal axes of the object (for static object orientation), and its minimum bounding box is computed to obtain dimensions and the center position. Finally, these unified 3D attributes are projected back to each frame via inverse transformation, ensuring the temporal consistency of frame-level annotations. This "fuse-then-backproject" strategy guarantees smooth temporal attribute estimations, avoiding jitters caused by frame-by-frame independent estimation.

Key Experimental Results¶

Main Results¶

The following presents the 3D detection results (AP3D@IoU=0.3) for the Car category on the KITTI validation set. All pseudo-labeling methods use MonoDETR as the downstream detector trained with the same configuration.

Method	Annotation Source	Extra Input	Easy	Moderate	Hard
MonoDETR (Fully Supervised)	GT-3D	-	79.72	65.87	58.83
OVMono3D (Open-Vocabulary)	GT-3D	-	74.01	51.25	42.40
3D-MOOD (Open-Vocabulary)	GT-3D	Depth	81.97	64.16	54.36
OVM3D-Det (Pseudo-Label)	GSAM	GPT-4 Dimension Prior	44.48	33.29	26.69
MonoSOWA (Pseudo-Label)	MViT2	Pose + Shape	72.70	56.30	47.70
PLOT (Ours)	GSAM	Video (20 frames)	80.48	60.83	51.49

On the KITTI-360 test set, PLOT likewise achieves the best [email protected] (Easy 54.78 / Hard 48.75), significantly outperforming MonoSOWA (42.72 / 46.59) and VSRD (50.86 / 43.45) which requires known poses. In long-range scenarios (>30m) on Waymo, PLOT's APBEV even surpasses the fully supervised MonoDETR, demonstrating its reliability under long distances and challenging lighting conditions.

Ablation Study¶

Importance of Global Object Memory (KITTI validation set Car, [email protected]):

Training Set	GOM	Easy	Moderate	Hard
KITTI	✗	59.82	51.00	45.40
KITTI	✓	80.48	60.83	51.49
KITTI-360	✗	48.14	-	40.28
KITTI-360	✓	54.78	-	48.75

Impact of Frame Count in Trajectory-Guided Shape Fusion (KITTI Car, [email protected]):

Frames Used	Easy	Moderate	Hard	Time per Frame
1 (Single Frame)	51.75	37.48	30.63	0.46s
+2 frames (3 frames total)	62.41	48.06	40.31	0.66s
+8 frames (9 frames total)	77.23	56.16	48.43	1.56s
+20 frames (21 frames total)	80.48	60.83	51.49	2.45s

Key Findings¶

Shape fusion is the largest contributor: Moving from single-frame to 20-frame fusion, the mean [email protected] improves by roughly 1.7 times, indicating that multi-view geometric information is crucial to compensate for single-frame observation limitations.
GOM is highly effective but scene-dependent: On KITTI, GOM brings an improvement of about 10-20 AP (KITTI scenes are relatively simple but have high detection noise), and about 6-8 AP on KITTI-360 (broader scenes and more challenging long sequences). The core value of GOM lies not in complex visual reasoning, but in eliminating short-term volatility in detection noise using temporal consistency rules.
PLOT as a standalone detector: The raw, untrained pseudo-labels (directly fitting bounding boxes to point clouds) achieve a Moderate [email protected] of approximately 51 on KITTI, outperforming OVM3D-Det (33.29) and the weakly supervised method MonoGRNet (42.61). This indicates that PLOT can generate usable 3D annotations even without training a downstream model.
Pedestrian verification demonstrates generalization: In pedestrian detection characterized by small scale and non-rigid motion, PLOT achieves a Moderate [email protected] of 14.39, far exceeding OVM3D-Det's 8.96. This proves that trajectory-guided fusion remains effective even without rigid body assumptions.
Motion guidance outperforms pure geometry in orientation estimation: Ablations show that incorporating camera motion estimation raises the mean AP by about 3 points, providing a highly reliable orientation prior especially for dynamic objects.

Highlights & Insights¶

Ingenuity of the Dual-Track Mask Design: The predicted masks (GSAM) capture actual frame-by-frame shape variations, while the tracked masks (AllTracker propagation) preserve point-level correspondences. These complementary lines provide both "geometric boundaries" and "correspondences" for subsequent motion estimation and shape fusion, which is far more robust than relying on a single mask source.
GOM is a Lightweight but Highly Effective Design: Without introducing any learnable parameters, it significantly repairs detection and tracking failures using only three rules (continuity check, attribute consistency check, and merging redundant entries), demonstrating the power of rule-based design.
"Fusion + Backprojection" Guarantees Temporal Consistency: Rather than estimating attributes independently per frame, it first fuses complete point clouds in the reference frame to extract unified attributes, and then projects them back to each frame. This simple "aggregate-then-distribute" strategy naturally ensures smooth annotations across frames.
The Optional Scale Term in Motion Estimation Displays Engineering Wisdom: Temporal scale drift in depth estimators does not occur in every frame. Having an optional activation flag avoids unnecessary degrees of freedom, striking a balance between flexibility and stability.
PLOT may be More Valuable as a Standalone Labeling Tool than a Training Data Provider: Experiments show that the raw pseudo-labels are already usable in zero-training scenarios, which is highly appealing for practical use cases that "only have video sequences without annotations" (such as automated labeling pipelines).

Limitations & Future Work¶

Dependency on Pre-trained Depth Estimators: Although the paper demonstrates robustness under noise, long-range depth noise remains the primary bottleneck; depth errors propagate directly to point cloud fusion and center estimation. The paper acknowledges significant center offset at distances >50m.
Limited Gains Under Near-Zero Relative Motion: When the relative motion between the camera and the object is near zero (e.g., vehicles driving slowly in the same direction far ahead), multi-view observations offer limited new perspectives, rendering shape fusion less advantageous.
Edge Cases in Association Failures: For objects observed too briefly near frame boundaries or heavily occluded objects, GOM may still fail to associate them correctly, leading to erroneous pseudo-point cloud fusion.
Computational Overhead Scales Linearly with Object Count: Generating labels takes about 2.8s per frame with a 20-frame window and 20 objects; real-time applications will require parallelization or lightweight versions.
Future Research Directions: Integrating stronger temporal continuity priors (such as long-term optical flow), incorporating multi-view geometric cues to enhance association in edge cases, or replacing rule-based GOM with learning-based association.

vs OVM3D-Det [CVPR 2024]: Operates on single frames and utilizes LLMs to estimate object dimension priors (e.g., "a car is typically 4.5m long"). PLOT utilizes video temporal information to fuse multi-frame observations, which not only yields more accurate shapes but also fully bypasses the need for LLM priors, avoiding uncontrollable semantic prior biases.
vs MonoSOWA [CVPR 2025]: Requires known camera intrinsics and IMU poses, relying on simplified mask-association. PLOT does not require any pose information (recovers relative motion through point tracking) and provides a more complete object geometry via GOM and trajectory-guided fusion, boosting KITTI AP by up to 9.68 points.
vs VSRD [ECCV 2024]: Pose estimation relies on multi-view and known poses, limited by static scene assumptions. PLOT can handle dynamic scenes and recovers poses from tracking rather than assuming them known, making it much more broadly applicable.
vs 3D-MOOD [CVPR 2025]: An open-vocabulary detector requiring large amounts of 3D training data. PLOT does not require any 3D training data under the zero-shot setting. In qualitative comparisons, 3D-MOOD misses objects in non-driving scenes where PLOT successfully produces annotations.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of dense point tracking + global memory + trajectory-guided fusion designed for pseudo-labeling is novel. The dual-track mask and GOM represent design combinations not seen prior to this.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Quantitative verification on three mainstream driving benchmarks, covering Car/Pedestrian/Vehicle; supplementary materials include extensive noise robustness analysis, failure mode analysis, and cross-domain qualitative comparisons.
Writing Quality: ⭐⭐⭐⭐ Clear structure, well-illustrated, and motivating narrative with rich layers; some technical details are compressed into supplementary materials, requiring flipping back and forth to compare tables in the main text, creating a slight reading burden.
Value: ⭐⭐⭐⭐⭐ Provides a low-cost 3D annotation path that genuinely requires no auxiliary sensors, offering direct practical value for autonomous driving data scaling, general scenery annotation, and weakly-supervised training.