Point2Pose: Occlusion-Recovering 6D Pose Tracking and 3D Reconstruction for Multiple Unknown Objects Via 2D Point Trackers

Conference: ECCV2026
Paper: ECCV 2026 / Project Page
Area: 3D Vision
Keywords: 6D pose tracking / multi-object tracking / occlusion recovery / 2D point tracking / TSDF reconstruction

TL;DR

Point2Pose turns a long-range 2D point tracker into a persistent data-association engine: a set of query points is scattered on each object, lifted to 3D with depth and accumulated into an object-centric keypoint map, and the pose follows from frame-to-map registration. This lets it track multiple unknown rigid objects simultaneously without any CAD model and recover immediately once an object reappears after complete occlusion — at the cost of slightly lower single-object ADD and reconstruction accuracy than BundleSDF.

Background & Motivation

Recovering the 6D pose of objects from monocular RGB-D is a core operation for robotics and AR, and a precondition for grasping, planning and manipulation. Today's strongest methods all rest on some form of "known object" assumption: PVN3D and FoundationPose rely on object CAD models, category-level methods rely on class priors, and another family demands a multi-view reconstruction before tracking can start. A robot in the open world can satisfy none of these — it faces objects it has never seen, and nobody has built the models in advance. The model-free route is therefore more attractive, but its representatives each have a hard limit: BundleTrack uses graph-based optimization with temporal feature matching to segment and track an object, BundleSDF swaps in a neural implicit field that jointly optimizes pose and shape, and 6DOPE-GS uses Gaussian Splatting for rendering speed and detail. Two problems are common to all of them. They are designed almost exclusively for a single object, and they cannot come back once the object is fully occluded — BundleSDF represents the object better, but the cost of neural-field optimization makes it hard to maintain several objects at once, and re-localization after prolonged complete occlusion remains difficult.

The core tension here is not representational capacity but data association itself. Tracking methods are smooth and cheap because they exploit inter-frame motion continuity; the moment an object is completely hidden by a robot arm, another object, or a camera that briefly moves away, frame-to-frame matching has nothing left to match against, and the system must fall back on a heavyweight, brittle global re-localization module — which additionally has to answer "which object just came back?". Yet multi-object manipulation is exactly where such complete occlusion is densest: arms sweep across the lens, objects cross over each other, the camera briefly leaves the workspace. The tension thus becomes: the more a method depends on frame-to-frame matching, the less it can recover from complete occlusion — and complete occlusion is the only situation where recovery genuinely matters.

This paper's angle is to turn "correspondence" from a one-shot inter-frame match into an object that persists across the whole sequence. A long-range 2D point tracker maintains the identity of every query point across frames: during occlusion the tracker merely marks those points as non-visible rather than discarding them, and when the object returns, the very same query point emits a pixel location again. Identity is never lost, so recovery needs no re-localization module. Lifting these points to 3D with depth and rotating them into the object frame yields a keypoint map that grows as the object turns, and the pose reduces to a registration problem between the current frame's observations and that map. Core idea: let a long-range 2D point tracker carry all data association, lift the tracked points to 3D with depth into an object-centric keypoint map, and solve the pose by frame-to-map registration — occlusion recovery then stops being a re-localization problem and becomes merely "waiting for the points to come back".

Method

Overall Architecture

The input is a monocular RGB-D video \(\{F_t\}_{t=0}^{T}\) with \(F_t=(I_t,D_t)\) in a fixed camera frame C; the goal is to output, for every frame, the 6D pose \(T^{C}_{O_i,t}\) of every tracked object \(i\), and at any later point to extract that object's triangle mesh from the online-fused TSDF via marching cubes. Each object frame \(O_i\) is initialized to coincide with the camera frame (\(T^{C}_{O_i,0}=I_{4\times4}\)), so all that remains is to estimate the increment \(T^{C}_{O_i,t}\). The system only ever asks the user to click a few points on the objects; SAM2 then produces a per-object instance mask, and no further human input is needed. There is no "reconstruct first, then track" staging either — the two are two faces of the same thing.

The pipeline is progressive. Object-specific 2D query points are sampled inside the masks and handed to a long-range point tracker, which tracks all objects' queries jointly and returns, per frame, a pixel location, a visibility indicator and an uncertainty score. Visible points are back-projected to camera-frame 3D observations with depth and rotated into the object frame using the current pose, accumulating into an object-centric keypoint map. The visible observations of the current frame are then registered against that map to yield the pose. When the object turns and reveals new surfaces, additional points are sampled and a keyframe is opened; on each keyframe an online factor graph is optimized and the segmented depth is fused into the object-centric TSDF. The TSDF here is not merely there to produce meshes: it is the dense geometric arbiter of the whole pipeline, used both to pick one pose among the candidates proposed by multi-hypothesis registration and to further refine the selected pose via SDF refinement.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB-D frames + user clicks"] --> B["SAM2 instance masks"]
    B --> C["Long-range point-tracker data association"]
    C --> D["Object-centric keypoint map<br/>sampling / pending promotion / expansion"]
    D --> E["Multi-hypothesis registration + TSDF refinement"]
    E --> F["Factor-graph optimization + TSDF reconstruction"]
    F -->|keyframe: pose and map feedback| D
    F --> G["6D pose trajectories + object meshes"]

Key Designs

1. Long-range point tracking as the data association: demoting occlusion recovery from re-localization to waiting

Existing model-free trackers (BundleTrack, BundleSDF) let frame-to-frame matching answer "which object does this pixel belong to", so their associations live for a single frame interval, complete occlusion severs them outright, and recovery requires an explicit global re-localization stage. Point2Pose answers the same question at a different level. The tracker's input is a set of query points \(q=(t_q,u_q)\), where \(t_q\) is the frame in which the point was queried and \(u_q\) its pixel location there; from then on the tracker returns, for every frame, the corresponding pixel location \(u_t\), a binary visibility indicator \(\delta_t\), and an uncertainty score \(\sigma_t\). A query point's identity persists across frames — meaning "which object this pixel belongs to" is fixed the moment the point is sampled and never has to be re-decided. When occlusion happens, the tracker only sets \(\delta_t\) to 0 while the point stays alive in the query set. Registration simply filters out the points with \(\delta_t=0\) and back-projects the rest using depth to obtain the valid correspondences of the current frame.

The direct consequence is that recovery becomes free: when the object reappears the tracker resumes emitting pixel locations for the same points, frame-to-map registration immediately has correspondences to work with, and no additional re-localization network or candidate search is required. Multi-object tracking falls out of the same mechanism: each object keeps its own query set \(Q_i\), but all query points are tracked jointly in a single pass, so object identity is carried by the grouping of the query sets and no cross-frame data association in the camera frame is needed at all. The price is that the system is now bound to the quality of the point tracker — which is precisely why the texture-poor sequences fail later on.

2. The object-centric keypoint map: which points to sample, and when a point counts

The handful of points sampled in the first frame get occluded or rotate out of view one after another, so the map must keep adding points to sustain tracking. Adding points has two opposite failure modes: selecting purely by trackability yields a cluster of points packed onto one patch of texture, and pose estimation then degenerates in the rotational direction; selecting purely for spread yields points sitting on textureless regions that cannot be tracked at all. The paper therefore runs SuperPoint inside the mask, treats the detector confidence \(s_k\) directly as a proxy for trackability, and greedily picks \(K\) points, at each step choosing the candidate that maximizes

\[J_k=\lambda\,s_k+(1-\lambda)\min\!\left(\frac{d_k}{r_{\text{ideal}}},1\right)-\beta\,\rho(d_k),\qquad \rho(d_k)=\max\!\left(0,\frac{r_{\min}-d_k}{r_{\min}}\right)\]

where \(d_k\) is the minimum distance from the candidate to the union of the already-selected points and the points already in the map, \(r_{\text{ideal}}\) is the ideal spacing radius, \(r_{\min}\) the minimum acceptable separation, and \(\lambda\in[0,1]\) and \(\beta\) are weights. The intent of the three terms is plain: the first rewards trackability, the second encourages spread until the spacing reaches \(r_{\text{ideal}}\), and the third hard-penalizes candidates that sit too close to existing tracks. ⚠️ The PDF extraction of Eq. (1) in the original paper is lossy (symbols and sub/superscripts are damaged); the form and symbol conventions here should be checked against the original paper and its appendix.

The newly selected \(K\) points are back-projected with depth and rotated into the object frame with the current pose, but they do not enter the map immediately: depth noise and pose error can both push a point's 3D coordinate off, and once such a bad point is written into the map it contaminates every subsequent registration. New points therefore start in a pending state, receive a score of +1 or +0 per time step based on the new observation, and are promoted to full keypoints only when the accumulated score exceeds a threshold within a fixed time window (the window length and threshold are given in the appendix, which is not available in the cache — ⚠️ refer to the original paper). The map is thus monotonically expanding: every newly revealed surface leaves behind a batch of verified points, and the frame in which new points are sampled is by definition a keyframe, triggering a factor-graph optimization and a TSDF fusion. This "suspect first, then admit by multi-frame vote" mechanism is cheap, yet it keeps single-frame depth noise out of the map.

3. Multi-hypothesis frame-to-map registration with the TSDF as arbiter

Given the correspondences, the pose is a standard least-squares registration problem: with the current frame's 3D keypoint observations \(\tilde p_n\) and their corresponding map points \(p_n\), solve

\[\hat T=\arg\min_{T\in SE(3)}\sum_{n=1}^{N}\left\|\tilde p_n-Tp_n\right\|^{2},\]

which admits a closed-form SVD solution. The problem is that 2D tracking produces a high ratio of outliers under severe occlusion or repetitive texture, and a single least-squares fit over all correspondences — or a single RANSAC pass — yields catastrophic poses. The paper's example is concrete: for a mug rotating about its vertical axis, the body is a rotationally symmetric cylinder, the tracker predicts nearly static trajectories for keypoints on it, and only the few points on the handle actually reflect the rotation. RANSAC's consensus set is then dominated by the numerically superior but wrong body points, and the correct solution is squeezed out.

The paper's answer is to let dense geometry decide who is right. First, a set of pose hypotheses is generated by sequential RANSAC paired with SVD: each iteration finds the transformation with the most inliers as usual, but then that consensus set is removed from the correspondence pool and the procedure continues on what remains. The candidate set therefore contains not only the dominant motion but also viable solutions that the dominant motion had suppressed, and these typically correspond to a genuine second rigid-body motion. Each hypothesis is then scored: the dense points from the current frame's masked depth image are transformed into the object frame under that hypothesis, their absolute TSDF values are read, and the hypothesis with the smallest mean |TSDF| wins. Sparse tracks propose, dense geometry disposes — the essence of symmetric-object degeneracy is that most correspondences carry no rotational information, whereas the TSDF is a global geometric prior independent of texture, where a few correct correspondences can overturn a large number of wrong ones. Finally, starting from the selected hypothesis, SDF refinement iteratively solves for a pose increment \(D\) that drives the current frame's point cloud onto the zero-level isosurface of the TSDF:

\[D^{*}=\arg\min_{D\in SE(3)}\sum_{p\in\mathcal P_{\text{cur}}}\rho_H\!\left(\Phi\!\left(T_i^{-1}D^{-1}p\right)\right),\qquad T_{i+1}=D^{*}T_i,\]

where \(\rho_H\) is the Huber robust kernel, the TSDF spatial gradients are computed numerically, and the increment is obtained with Levenberg–Marquardt over \(L\) iterations. The Huber kernel matters here: even after hypothesis selection, residual outliers would still drag the refinement result off through their quadratic terms.

4. Online factor-graph optimization and object-centric TSDF reconstruction

Frame-to-map registration is independent per frame, so its errors accumulate into drift; and the map's keypoints are themselves not clean (depth noise; the promotion threshold filters rather than eliminates it). Hence, at every new keyframe, the object's keyframe poses and keypoint coordinates are jointly optimized. To obtain a textbook landmark observation model, the authors write the problem in the equivalent "fixed object observed by a moving virtual camera" view and parameterize it with inverse pose variables \(X_m:=T^{O_m}_{C}\); the objective has three terms, \(\mathcal L=\mathcal L_{\text{prior}}+\mathcal L_{\text{reg}}+\mathcal L_{\text{obs}}\). The prior anchors the first keyframe pose to the identity, purely to remove gauge ambiguity so that the whole graph cannot drift while leaving the objective unchanged. The pose consistency term penalizes keyframe poses deviating from the frame-to-map registration results, but it uses relative motion constraints rather than absolute pose constraints, and it allows each keyframe its own registration covariance \(\Sigma_{\text{reg},m}\) — this matters, because per-frame registration confidence genuinely varies, and the result on a low-texture frame should not be trusted as much as one on a crisp frame. The observation term measures the discrepancy between a map keypoint \(p_n\) projected into that keyframe's camera frame via \(X_m^{-1}\) and the measured 3D point \(\tilde z_{m,n}\), and does so in a bearing–range (spherical) representation: the 3D point is mapped to a unit direction and a radius, and the angular difference and the distance difference are compared separately, both under a Huber kernel. This keeps directional error and depth error from diluting each other inside a single Euclidean distance. The whole nonlinear least-squares problem is solved with GTSAM's Levenberg–Marquardt; the factor graph is updated and re-optimized at every keyframe insertion, and the optimized poses and keypoint coordinates are fed back to both tracking and reconstruction.

Reconstruction shares the same geometry. On each keyframe the segmented RGB-D observations are fused, by a projective TSDF update rule, into a volumetric grid defined in the object frame, updated online only when a keyframe arrives; when a mesh is needed, marching cubes is applied and disconnected components caused by depth noise are filtered out. Notably, this TSDF volume serves three purposes at once: the final mesh, the scoring function for hypothesis selection in multi-hypothesis registration, and the implicit surface used by SDF refinement. Reconstruction and tracking are therefore not two pipelines but two uses of one representation. This is also why the method can "track and build" without CAD: the map needs only a few points from the first frame to start, with no need to wait for a clean mesh before tracking.

A Worked Example: a mug rotating about its vertical axis

Suppose that in the first frame the user clicks three points on a mug and three on a tomato soup can, and SAM2 produces two instance masks.

Step one: inside the mug's mask, SuperPoint reports a batch of candidates, and \(K\) reasonably spread, high-confidence points are greedily selected by \(J_k\). Together with their query frame indices they form the mug's query set \(Q_0\); the can gets \(Q_1\) the same way. Both sets are tracked jointly in a single pass, so "does this point belong to the mug or the can" never has to be re-decided from this moment on.

Step two: the mug starts to rotate. Its body is a rotationally symmetric cylinder, so points on the body barely move, and only the few points on the handle truly reflect the rotation. With a single RANSAC pass, the numerically dominant body points would yield a consensus solution of "barely rotated". Sequential RANSAC keeps iterating on the remaining points, so the handle-consistent solution survives as a second hypothesis; the current TSDF then scores both — the dense depth points of this frame are transformed into the object frame under each hypothesis and their |TSDF| is read. The handle solution scores clearly lower, so it is selected and then driven onto the zero-level isosurface by SDF refinement.

Step three: the mug turns far enough to reveal a new surface, which opens a new keyframe. A fresh batch of points is sampled there and starts in the pending state; only after several consecutive time steps of consistent observation are they promoted into the map. A factor-graph optimization then re-optimizes keyframe poses and map points together, and this frame's depth is fused into the mug's TSDF.

Step four: the soup can fully occludes the mug for a number of frames. The tracker sets the body points' visibility to 0 without discarding them; registration filters those points out wholesale, so the pose is carried by the few points still visible — and simply extrapolated when there is nothing to observe at all. Once the can moves away, the same query points reappear inside the mask, the tracker emits their pixel locations directly, and frame-to-map registration restores the pose immediately, with no re-localization search. The paper reports that on two occlusion-heavy real sequences Point2Pose recovered 5/5 complete-occlusion events within 30 frames of the object re-emerging, whereas P2P-SH(10) — a variant that permanently discards a point after 10 consecutive invisible frames — recovered 0/5.

Loss & Training

There is no training stage; this is a purely optimization-based system. SAM2, SuperPoint and the 2D point tracker (Track Any Points or the CoTracker family) are all off-the-shelf models, kept frozen and never fine-tuned. What has to be solved is only the closed-form SVD of frame-to-map registration, the iterative nonlinear least-squares of SDF refinement, and the LM optimization of the factor graph, the latter implemented in GTSAM. The "hyper-parameters" are therefore mainly the sampling and mapping knobs — \(K\), \(r_{\text{ideal}}\), \(r_{\min}\), \(\lambda\), \(\beta\), plus the pending points' verification window and promotion threshold. Their concrete values are given in the appendix, which is not available in the cache (⚠️ refer to the original paper), and the main text reports no sensitivity analysis for them. Runtime is 2–10 Hz depending on tracker resolution and the number of tracked points; the bottleneck is squarely the 2D point tracker, whose cost scales roughly linearly with the number of tracks. The authors stress that the system is modular with respect to the point tracker and that their Python implementation is not runtime-optimized, so both a faster tracker and engineering work would directly cut latency.

Key Experimental Results

Experiments cover three benchmarks: two public datasets — the HO3D_v3 evaluation split (13 hand–object interaction sequences, 4 objects) and YCBInEOAT (9 dual-arm robot manipulation sequences, 5 YCB objects, objects fairly small in the image, poses manually annotated) — plus the newly introduced YCBMultiTrack. Two baselines are compared: BundleSDF (CAD-free, and hence model-free like this paper) and FoundationPose (given the objects' CAD models). Pose is measured by the AUC of ADD and ADD-S over the 0–0.1 m threshold range, where ADD is exact pose error and ADD-S accommodates object symmetry via closest-point matching; reconstruction quality is measured by Chamfer distance (CD, in cm) between the reconstructed and ground-truth meshes. YCBMultiTrack has a synthetic and a real part. The synthetic part is rendered in Isaac Lab with 7 YCB objects, covering one single-object and three two-object scenarios, with objects moving along linear or circular trajectories. The real part is captured with an Intel RealSense D435i over 5 YCB objects, covering five single-, four two- and two three-object scenarios, with pose ground truth from an OptiTrack motion-capture system (four to five markers per object), camera extrinsics calibrated with an AprilTag of known ground-truth pose, and optimization-based time synchronization.

Main Results

Tables 1 and 2 give the mean comparison on the single-object public benchmarks (on HO3D, FoundationPose fails on the SM1 sequence and the mean is taken over the remaining 12).

Dataset	Metric	FoundationPose (with CAD)	BundleSDF (CAD-free)	Point2Pose (CAD-free)
HO3D_v3	ADD-S AUC (%) ↑	97.16	93.34	94.63
HO3D_v3	ADD AUC (%) ↑	93.40	87.36	80.79
HO3D_v3	CD (cm) ↓	—	0.58	1.02
YCBInEOAT	ADD-S AUC (%) ↑	96.00	94.51	92.67
YCBInEOAT	ADD AUC (%) ↑	92.48	89.07	85.11

Table 3 covers YCBMultiTrack. Because both baselines support only single-object tracking, they are run sequentially, one object at a time, on the multi-object sequences.

Split	Metric	FoundationPose (with CAD)	BundleSDF (CAD-free)	Point2Pose (CAD-free)
Synthetic	ADD-S AUC (%) ↑	98.39	76.74	88.67
Synthetic	ADD AUC (%) ↑	97.58	63.05	77.94
Real	ADD-S AUC (%) ↑	42.49	62.08	89.43
Real	ADD AUC (%) ↑	35.23	42.34	74.17

Ablation Study

The component-wise ablation is run on HO3D, removing one module at a time.

Config	ADD-S AUC (%) ↑	ADD AUC (%) ↑	Note
Full model	95.07	82.76	full model
w/o multi-hypothesis	89.81	65.52	replaced by one SVD alignment + fixed-threshold outlier removal + a second SVD; largest drop (ADD −17.24)
w/o SDF refinement	94.40	77.37	uses the pose selected by the multi-hypothesis stage directly (ADD −5.39)
w/o graph optimization	94.56	78.80	no graph-based global pose refinement (ADD −3.96)

Replacing long-range tracking with short-horizon tracking is the ablation that directly tests the central hypothesis: P2P-SH(10) permanently discards a point after 10 consecutive invisible frames, with every other module unchanged.

Config	Occlusion events recovered	mustard seq. ADD-S / ADD AUC (%)	Note
Point2Pose	5/5 (within 30 frames of re-emergence)	93.79 / 84.36	query-point identity kept throughout
P2P-SH(10)	0/5	38.52 / 31.44	points discarded for good, so recovery is impossible

⚠️ The "Full model" row of the ablation table (95.07 / 82.76) does not match the HO3D mean in Table 1 (94.63 / 80.79); the ablation presumably uses a different sequence subset or evaluation configuration, so refer to the original paper, and the two tables' numbers should not be compared directly against each other.

Key Findings

• Multi-hypothesis registration is the largest single contributor among the three modules: removing it costs 5.26 ADD-S and 17.24 ADD, far more than removing SDF refinement (−5.39 ADD) or graph optimization (−3.96 ADD). This matches the paper's stated motivation — outliers and symmetric degeneracy are the dominant failure sources for model-free tracking, and both refinement and global optimization presuppose that the hypothesis was chosen correctly.
• Occlusion recovery comes entirely from the persistence of query-point identity: truncating a point's lifetime to 10 frames drops the recovery rate from 5/5 to 0/5 and collapses the same mustard sequence's ADD-S AUC from 93.79 to 38.52. This is the cleanest piece of causal evidence in the paper.
• The method trades single-object accuracy for a broader model-free capability: on HO3D, ADD-S AUC is slightly above BundleSDF (94.63 vs 93.34), but ADD is clearly lower (80.79 vs 87.36) and reconstruction CD is worse (1.02 vs 0.58 cm). High ADD-S with low ADD means the pose is roughly aligned while the orientation-sensitive part of the accuracy degrades — the classic symptom of symmetric ambiguity compounded by point-tracking noise.
• The failure mode concentrates on low-texture surfaces: on the HO3D AP12 sequence ADD is only 46.64 (BundleSDF 94.54), on MPM13 33.71 (BundleSDF 63.19), and on YCBInEOAT bleach v1 ADD-S is 82.88 (BundleSDF 94.00). Using Sobel gradient magnitude to quantify local texture, the authors project ground-truth keypoints and compare them with the tracker's predictions, finding that weaker texture means larger 2D point-tracking error and lower downstream ADD AUC. The failure chain is thus "low texture → unreliable long-range correspondences → degraded frame-to-map registration", not a problem in reconstruction or optimization. Notably, ADD-S on these sequences often remains decent, indicating that the pose is at least coarsely aligned.
• Multi-object plus complete occlusion is where the method shines: on YCBMultiTrack-Real, FoundationPose — despite having CAD meshes — collapses to 42.49 ADD-S / 35.23 ADD, and BundleSDF fails or degrades widely (62.08 / 42.34), while this method holds 89.43 / 74.17. The paper's qualitative comparison shows BundleSDF tracking correctly before the object leaves the view, then initializing its re-localization from that stale estimate once the object re-enters, so the pose comes back wrong.
• On the synthetic split the CAD-based baseline still wins: FoundationPose leads YCBMultiTrack-Synthetic at 98.39 / 97.58. The authors explain that objects stay largely continuously visible in the synthetic sequences, so long-term occlusion recovery is not truly stressed, and such a setting therefore does not showcase the method's advantage.

Highlights & Insights

• Turning data association from a one-shot match into a persistently existing query identity: this is the most elegant step in the paper. It reframes occlusion recovery from "re-localization" into "waiting", at the cost of only a point tracker's memory and compute. Any pipeline that currently relies on frame-to-frame matching for long-term tracking (landmark management in visual SLAM, multi-object tracking, long-horizon manipulation) could consider swapping that layer for a long-range point tracker, with recovery coming almost free.
• Sparse proposes, dense disposes: a few outlier-laden tracked points generate a set of candidate poses via sequential RANSAC, and dense TSDF scoring picks one. The combination works because symmetric-object degeneracy is fundamentally "most correspondences carry no rotational information", while the dense geometric prior does not depend on texture — so a few correct correspondences can overturn the majority, with no learned symmetry prior needed.
• The keypoint map is simultaneously the "model" and the registration target: model-free tracking usually has to reconstruct the object before tracking it, whereas here the map and the pose enable each other — the map supplies correspondences for registration, and the registration result places new points in the object frame. This is what allows a cold start from a few points in the first frame and tracking-while-building.
• The pending state is a remarkably cheap noise gate: never trust a single-frame 3D observation, and let a new point prove itself by multi-frame voting before entering the map. The same pattern transfers directly to any online mapping system that continuously ingests 3D observations from a moving sensor, blocking depth noise at the door instead of cleaning it up afterwards.
• The dataset fills a genuine evaluation gap: HO3D / YCBInEOAT are single-object, YCB-Video is static multi-object, and HOT3D is multi-view RGB without dense depth. YCBMultiTrack supplies dynamic multi-object motion, inter-object occlusion, and — crucially — per-object visibility annotations (whether an object is currently visible or fully occluded). That last item is something almost no prior benchmark could measure directly, and without it "occlusion recovery" cannot be quantified at all.

Limitations & Future Work

• The authors admit four limitations: the underlying 2D point tracker relies on sufficiently discriminative image texture and can become unreliable on textureless or repetitive surfaces; the method depends on accurate instance segmentation, and noisy or incomplete masks introduce background points that contaminate registration; tracking many objects increases memory usage through the sheer number of tracked points (mitigable by point subsampling or pruning inactive keypoints); and the classical TSDF representation prioritizes simplicity and runtime efficiency at the cost of reconstruction fidelity relative to recent learning-based approaches, with "introducing neural reconstruction (e.g. EfficientNeRF) while keeping online fusion and tight coupling with tracking" named as future work.
• Several things, in my reading, are left unaddressed by the experiments. First, the knobs that directly govern the accuracy/memory/latency trade-off — the number of sampled points \(K\), the ideal spacing \(r_{\text{ideal}}\), the minimum separation \(r_{\min}\), and the pending promotion threshold — get no sensitivity analysis in the main text (their details sit in an appendix unavailable in the cache), so a reader cannot judge how hard the system is to tune. Second, occlusion recovery is quantified only as 5/5 on two real sequences; there is no distribution of recovery latency across all sequences, and no report of how the system degrades when recovery fails (does it hold the stale pose, or jitter?). Third, object count and identity still require manual clicking in the first frame; the system will not automatically discover a new object entering the scene, which is hardly rare in multi-object manipulation. Fourth, in the multi-object scenarios both baselines are run sequentially per object — the authors honestly note this gives the baselines a slight computational advantage — but the baselines equally cannot exploit cross-object information, and no compute-normalized comparison is given. Fifth, the HO3D reconstruction CD degrades from 0.58 cm to 1.02 cm (2.97 cm on AP12); the paper attributes this only to texture, but it cannot be ruled out that map expansion keeps depositing noisy depth points into the TSDF with additions but no eviction.
• Concrete improvement directions: replace the raw depth used for lifting points with a depth foundation model or multi-view stereo, directly reducing sensitivity to depth noise; replace "trackability" as proxied by SuperPoint confidence with the tracker's own \(\sigma_t\) or a small learned predictor; give the keypoint map an eviction mechanism (by informativeness rather than visibility alone) so stale bad points cannot occupy the map indefinitely; model symmetry explicitly rather than relying only on multi-hypothesis fallback; and the neural-reconstruction swap the authors themselves propose.

Related Work & Insights

• vs BundleSDF: both are model-free, both track and reconstruct online, and neither needs CAD. BundleSDF represents the object with a neural implicit field jointly optimized with the pose and maintains associations through frame-to-frame matching; this paper represents it with a sparse keypoint map plus a TSDF (cheaper, which is what makes several objects tractable) and replaces frame-to-frame matching with persistent long-range point queries. The price is lower reconstruction fidelity (HO3D CD 1.02 vs 0.58 cm) and lower orientation accuracy on texture-rich objects (ADD 80.79 vs 87.36); the payoff is multi-object support and genuine recovery from complete occlusion.
• vs BundleTrack: also model-free, also a graph-optimization single-object tracker with temporal feature matching. The difference is that its correspondences are short-horizon frame-to-frame matches, so it drifts and fails outright under complete occlusion, whereas here correspondences are long-range query identities that are merely marked non-visible during occlusion.
• vs FoundationPose: the strongest tier in accuracy, but it requires object CAD models. Interestingly, in the occlusion-heavy real multi-object setting of YCBMultiTrack-Real it collapses to 42.49 ADD-S even when handed the CAD mesh — evidence that the bottleneck for occlusion recovery is data association, not object priors. A further detail is that it fails outright on HO3D's SM1 sequence.
• vs 6DOPE-GS: uses Gaussian Splatting for real-time tracking and reconstruction, with better rendering speed and detail, but it still cannot re-localize after the target completely leaves and re-enters the view; the problem domains are complementary.
• vs CosyPose / KMOPS: both provide poses for multiple objects in a scene, but they target static objects under multi-view or stereo input rather than persistent temporal tracking. This paper handles continuous 6D trajectories of multiple dynamically moving objects, including inter-object crossovers and temporary disappearance.
• Dataset lineage: HO3D and YCBInEOAT are single-object manipulation; YCB-Video is multi-object but essentially static (only the camera moves) and cannot capture independent object trajectories or inter-object occlusion; HOT3D is large-scale egocentric multi-object interaction but with multi-view RGB and monochrome streams only, lacking dense depth. YCBMultiTrack targets exactly the combination of "dynamic multi-object + RGB-D + complete occlusion + motion-capture ground truth + per-object visibility annotations".

Rating

• Novelty: ⭐⭐⭐⭐ Replacing frame-to-frame matching with a long-range point tracker to carry data association, thereby turning occlusion recovery into "waiting for the points", is a clean and broadly applicable change of perspective; multi-hypothesis plus TSDF arbitration is sound but comparatively engineering-driven.
• Experimental Thoroughness: ⭐⭐⭐ Three datasets, both simulation and real, plus occlusion-event and texture analyses are covered, but sensitivity analyses for the sampling and promotion thresholds are missing, the ablation table's HO3D numbers disagree with the main table, and recovery latency is quantified only on a subset.
• Writing Quality: ⭐⭐⭐⭐ Well structured, with complete module and formula exposition, and honest in separating the "with CAD" and "CAD-free" settings and in stating outright that single-object accuracy is traded for a broader model-free capability.
• Value: ⭐⭐⭐⭐ Model-free multi-object tracking with recovery from complete occlusion is a hard requirement in robotic manipulation; the system is modular with respect to the point tracker (so it benefits directly from tracker progress), and code and dataset are promised to be released.