Where and What: Long-Term Object Tracking in Egocentric Videos¶

Conference: ECCV 2026
Paper: ECCV Original
Code: https://jacobchalk.github.io/Whareformer/
Area: Video Understanding
Keywords: Egocentric Video, Long-Term 3D Object Tracking, Object Permanence, Transformer Association, Spatiotemporal Memory

TL;DR¶

Whareformer presents the first learning-based framework for the Out of Sight, Not Out of Mind (OSNOM) task in egocentric videos, projecting relative 2D appearance and 3D spatial distances into learned likelihood embeddings and employing an explicit New Track token within a Transformer encoder to achieve 72.2% mPCL and 84.0% IDF1 on EPIC-KITCHENS-100.

Background & Motivation¶

Humans naturally maintain a cognitive mental map of their physical surroundings, effortlessly keeping track of multiple objects of interest—such as keys left on the kitchen counter, a smartphone on the coffee table, or a mug by the sink. Even when items are temporarily occluded by hands, placed behind other objects, or completely out of sight due to body rotation, humans retain a robust sense of "what is where." For an autonomous embodied agent operating in human environments, replicating this spatial cognitive capacity and object permanence across long temporal horizons is essential. To formalize this goal in egocentric videos, recent research introduced the Out of Sight, Not Out of Mind (OSNOM) task, evaluating an online model's ability to recall 3D object positions even when they leave the field of view and maintain instance identities across continuous manipulation and viewpoint shifts.

However, prior egocentric 3D tracking methods (such as LMK and IT3DEgo) rely predominantly on handcrafted association metrics and Hungarian matching governed by rigid, manually calibrated thresholds. Under rapid head motion, frequent object manipulation, or in cluttered environments where objects are repositioned into spaces previously occupied by others, fixed heuristics struggle to adaptively balance appearance (what) versus spatial distance (where). Crucially, heuristic frameworks treat the creation of a new track as a passive fallback triggered only when all matching costs exceed a threshold, which impairs the system's ability to deliberately decide when an observation belongs to an entirely new object.

This paper addresses these limitations by replacing heuristic rules with a trainable, data-driven association and memory pipeline. Core idea: project the relative appearance and 3D metric distances between current observations and established tracks into learned likelihood embeddings, and introduce an explicit New Track token to frame new track creation as an active, competitive decision within a permutation-invariant Transformer encoder.

Method¶

Overall Architecture¶

Whareformer operates online on incoming egocentric video streams. For each detected object mask at frame $t$, the system extracts an observation $o_n^t = (a_n^t, l_n^t)$, where $a_n^t$ denotes a PCA-reduced and $\ell_2$-normalized DINOv2 appearance descriptor, and $l_n^t$ represents the object centroid lifted into the reconstructed 3D world coordinate frame using aligned monocular depth. The model maintains a memory of $T$ active tracks $\mathcal{T} = \{T_1, \dots, T_T\}$. For each observation, Whareformer measures its relative Euclidean distance to the appearance clusters and recent 3D location buffer of each existing track, projects these scalar distances into likelihood tokens via a learnable linear layer, prepends a dedicated learnable NT (New Track) token, and processes the permutation-invariant sequence through a single Transformer encoder layer. A linear classification head then outputs a probability distribution over all existing tracks and the new track option, resolving multi-observation conflicts via confidence-based greedy assignment before adaptively updating the track memory.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current Observation<br/>DINOv2 Appearance + 3D Coordinates"] --> B["Relative Distance Computation<br/>Min Appearance Dist + Min Spatial Dist"]
    M["Dynamic Track Memory<br/>DenStream Clusters + 1s Buffer"] --> B
    B --> C["Learnable Projection & Embeddings<br/>Feature-Agnostic Distance Mapping"]
    C --> D["New Track Token Concatenation<br/>[NT, e_1, ..., e_T] Unordered Sequence"]
    D --> E["Transformer Association Encoder<br/>Full Cross-Token Feed-Forward Attention"]
    E --> F["Confidence Conflict Resolution<br/>Greedy Highest-Confidence Assignment"]
    F --> G["Adaptive Track Memory Update<br/>DenStream Streaming Update + Buffer Push"]

Key Designs¶

1. Relative Distance Extraction and Feature-Agnostic Likelihood Embedding: Decoupling Generalization from Raw Features
Conventional trackers feeding high-dimensional visual feature vectors or absolute coordinates directly into matching networks frequently overfit to specific object categories or room layouts. Whareformer abstracts the matching problem into relative metric distances: for observation $o_n$ and track $T_k$, it calculates the minimum squared Euclidean distance to the track's appearance cluster centers $c_{n,k}^A$ and the minimum Euclidean distance to its recent 3D spatial buffer $c_{n,k}^L$: $$c_{n,k}^A = \min_{a' \in \mathcal{A}(T_k, t)} \text{dist}(a_n, a')^2, \quad c_{n,k}^L = \min_{l' \in \mathcal{L}(T_k, t)} \text{dist}(l_n, l')$$ These two scalar distances are concatenated and mapped via a trainable linear projection $g: \mathbb{R}^2 \to \mathbb{R}^d$ to produce an assignment likelihood embedding $e_{n,k} = g([c_{n,k}^A, c_{n,k}^L])$. Because $e_{n,k}$ encodes geometric and feature separations rather than raw sensory attributes, the network learns a transferable trade-off surface that generalizes across unseen object classes and distinct environments.

2. Explicit New Track Token and Permutation-Invariant Transformer: Elevating Track Creation to a First-Class Decision
Heuristic approaches establish new tracks merely when all candidate matching costs surpass a fixed threshold, easily misassigning newly introduced objects to lingering old tracks under severe appearance changes. Whareformer prepends a learnable New Track token $\mathrm{NT} \in \mathbb{R}^d$ to the candidate embedding sequence: $$\mathbf{S}_n = [\mathrm{NT}, e_{n,1}, e_{n,2}, \dots, e_{n,T}] \in \mathbb{R}^{(T+1) \times d}$$ Because track indices have no natural ordering, the sequence omits positional encodings, ensuring rigorous permutation invariance. Inside the Transformer encoder, self-attention enables the $\mathrm{NT}$ token and all candidate track tokens to mutually contextualize their relative compatibilities. The linear head $h$ generates a normalized distribution $\hat{y}_n = \text{softmax}(h(\mathbf{Z}_n))$, unifying track association and track birth into a single forward pass. Simultaneous multi-observation conflicts within the same frame are handled by greedily accepting the highest-confidence assignment and iteratively assigning remaining observations based on subsequent scores, bypassing the higher computational overhead of Hungarian matching.

3. Dual-Track Memory Architecture: Balancing Long-Term Evolution and Spatial Locality
Manipulated objects undergo significant appearance transformations (rotation, state changes, occlusion) alongside large spatial relocations. Storing only the most recent observation causes catastrophic forgetting upon long-term re-entry, whereas an infinite history buffer incurs prohibitive memory and compute costs. Whareformer resolves this via an asymmetric memory design: - Appearance Memory $\mathcal{A}(T_k, t)$: Leverages the online DenStream density-based clustering algorithm to process the visual feature stream into persistent clusters (preserving long-term canonical appearances) and transient clusters (capturing short-term interaction variations). This yields an accurate approximation of an infinite visual buffer while bounding memory to $\approx 220$ MB and sustaining 322 FPS across long videos. - Location Memory $\mathcal{L}(T_k, t)$: Unlike appearance, retaining the complete spatial trajectory of an object creates ambiguity, as trajectories of different objects frequently intersect across shared surfaces. Hence, spatial memory is restricted to a sliding temporal buffer of length $W$ ($\approx 1$ second), capturing the object's immediate spatial position to filter sensor noise while preventing trajectory cross-talk.

Loss & Training¶

Whareformer is trained with teacher forcing on ground-truth track assignments. Batches are constructed by sampling observations across video frames, padded to the maximum track count within the batch, and supervised with standard cross-entropy loss: $$\mathcal{L} = -\sum_{c=0}^{T} y_c \log(\hat{y}_{n, c})$$ where index 0 corresponds to the $\mathrm{NT}$ token. To mitigate the distribution shift between ground-truth training tracks and model-predicted inference tracks, the authors apply DAgger-style data collection, re-extracting training samples using model predictions every 30 epochs and progressively increasing the ratio of self-generated trajectories, thereby boosting cross-dataset robustness.

Key Experimental Results¶

Main Results¶

On the EPIC-KITCHENS OSNOM test split (54 long videos, $\approx 12.2$ hours), as well as zero-shot transfer on IT3DEgo (HoloLens2, varied indoor rooms) and HD-EPIC (Aria Glasses, dense pick-and-place), Whareformer is evaluated against prior 2D and 3D baselines using mPCL (Percentage of Correct Locations within 30cm) and IDF1 (track identity preservation), as detailed in Table 1 of the paper.

Dataset	Metric	Whareformer (Ours)	LMK (Prev. SOTA) [26]	LMK-Inf (Infinite Memory)	Gain (vs LMK)
EPIC-KITCHENS	mPCL (%)	72.2	53.0	60.9	+19.2%
	IDF1 (%)	84.0	70.0	76.6	+14.0%
IT3DEgo	mPCL (%)	58.1	31.0	40.8	+27.1%
	IDF1 (%)	87.9	69.2	75.5	+18.7%
HD-EPIC	mPCL (%)	41.6	34.5	36.5	+7.1%
	IDF1 (%)	51.6	28.0	31.9	+23.6%

Ablation Study¶

Ablations on core architectural modules (Table 2 in the paper) and memory representation designs (Table 3 in the paper) on the EPIC-KITCHENS test set:

Config	mPCL (%)	IDF1 (%)	Memory (MB)	Speed (FPS)	Note
Whareformer (Full model)	72.2	84.0	220.7	322	Dual-modal + learnable NT + DenStream + 1s buffer
L-Only (w/o Appearance A)	34.0	50.5	-	-	Severe collapse when objects occupy former spots (-38.2% mPCL)
A-Only (w/o Location L)	62.6	79.5	-	-	Fails to disambiguate visually identical object instances (-9.6% mPCL)
No NT token (Threshold 0.25)	53.0	69.7	-	-	Reverting to threshold-based track birth causes sharp drop (-19.2% mPCL)
Fixed NT token (Non-learned)	72.0	83.9	-	-	Architecture structure drives the gain; learned weights add small boost
Appearance: 1-sec window	59.4	74.7	76.7	603	Short-term window drops identity on multi-minute re-entries
Appearance: Infinite memory	70.6	83.2	1843.8	50	8.4x memory expansion and noise accumulation degrade mPCL below DenStream

Key Findings¶

The explicit NT token is the single most critical structural component: Eliminating the $\mathrm{NT}$ token in favor of a 0.25 threshold triggers a 19.2% drop in mPCL (72.2% $\to$ 53.0%) and a 14.3% drop in IDF1 (84.0% $\to$ 69.7%), demonstrating that learned global competition between track creation and association is superior to heuristic thresholding.
DenStream achieves higher accuracy than unbounded infinite memory: By separating persistent modes from transient manipulation noise, DenStream outperforms infinite memory by +1.6% in mPCL while slashing memory usage from 1843.8 MB to 220.7 MB and boosting throughput from 50 to 322 FPS.
High data efficiency enables robust zero-shot cross-dataset transfer: Trained on only 56 videos from EPIC-KITCHENS, Whareformer achieves a +27.1% mPCL gain over LMK on the non-kitchen HoloLens2 IT3DEgo dataset, verifying that learning over relative distance manifolds avoids overfitting to specific sensor or scene distributions.

Highlights & Insights¶

Track initiation modeled as active token competition: Rather than handling track initialization as an ad-hoc exception after failed matching, Whareformer treats new track creation as an equal candidate token in an attention-driven decision space.
Distance-based projection as a domain-invariant inductive bias: Projecting distance scalars rather than raw feature embeddings provides effective regularization, allowing the model to generalize seamlessly across completely unseen object categories and recording hardware.
Streaming clustering as an efficient temporal memory: Deploying DenStream for evolving visual histories delivers a practical blueprint for maintaining long-term instance permanence in streaming embodied perception without memory explosion.

Limitations & Future Work¶

Dependency on upstream 2D segmentation and monocular lifting quality: Whareformer assumes externally provided 2D masks and lifted 3D centroids; catastrophic drift in visual odometry or monocular depth alignment degrades downstream tracking accuracy.
Lack of explicit modeling for topological object state transitions: Irreversible physical state transformations in culinary interactions (e.g., slicing an onion into rings or spreading butter) stretch smooth clustering assumptions, necessitating state-aware or part-based tracking representations.
Future Directions: Integrating Whareformer into an end-to-end embodied 3D dynamic scene graph pipeline to link long-term object permanence with robotic manipulation affordances.

vs LMK (3DV 2025): LMK uses hand-crafted distance weights, fixed heuristics, and Hungarian matching; Whareformer introduces an end-to-end learnable Transformer association framework with an explicit NT token, improving mPCL by +19.2% on EPIC-KITCHENS with faster inference.
vs IT3DEgo (CVPR 2024): IT3DEgo relies on privileged enrollment of object templates at frame 1 and assumes an oracle object count $M = K$; Whareformer solves the unconstrained online discovery and tracking setting without privileged object counts, outperforming IT3DEgo by +30.6% mPCL on IT3DEgo itself.
vs 2D Trackers (ByteTrack): 2D trackers fail during camera rotation and out-of-view intervals (achieving only 12.3% mPCL on EPIC-KITCHENS); Whareformer grounds objects in a consistent 3D world coordinate frame, maintaining object permanence throughout multi-minute absences.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Pioneering learnable Transformer association and dedicated NT token for the OSNOM benchmark with distance-based feature decoupling)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Thorough cross-dataset evaluations on EPIC-KITCHENS, IT3DEgo, and HD-EPIC with in-depth memory and runtime ablations)
Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous problem formulation, transparent design explanations, and compelling motivation)
Value: ⭐⭐⭐⭐⭐ (A fundamental milestone for long-term spatial memory and object permanence in first-person embodied vision)