FoundPose: Unseen Object Pose Estimation with Foundation Features

Conference: ECCV 2024
arXiv: 2311.18809
Code: Yes (evinpinar.github.io/foundpose)
Area: Human Understanding / Object Pose Estimation
Keywords: 6D Pose Estimation, DINOv2, Foundation Models, Training-free, Template Matching

TL;DR

FoundPose leverages a frozen DINOv2 foundation model to extract patch descriptors, establishing 2D-3D correspondences via bag-of-words template retrieval and kNN matching. It achieves zero-shot 6D pose estimation of unseen objects without any task-specific training, significantly outperforming existing RGB methods on BOP benchmarks.

Background & Motivation

Problem Definition

6D object pose estimation (3D rotation + 3D translation) is a core problem in spatial AI, with important applications in robotic manipulation and mixed reality. This paper focuses on model-based unseen object pose estimation: assuming a 3D mesh model is available, but there is no sufficient budget for large-scale dataset rendering and network training.

Limitations of Prior Work

Early Classical Methods (handcrafted feature matching, template matching): Can handle unseen objects, but have limited accuracy.

Deep Learning Methods (MegaPose, GigaPose, GenFlow): Highly accurate but require pre-training on massive task-specific datasets (millions of synthetic images, thousands of different objects), which makes dataset generation extremely costly.

Existing Training-Free Methods (ZS6D, PoMZ): ZS6D uses the last layer of ViT features to establish correspondences, achieving significantly lower accuracy than FoundPose; PoMZ requires RGB-D input.

Key Insight

Visual foundation models (e.g., DINOv2) exhibit strong generalization capabilities after self-supervised training. The authors discover that the intermediate layers' patch descriptors of DINOv2 carry both location information and semantic information, enabling reliable 2D-2D correspondences across the synthetic-to-real domain gap. This is a key insight: shallow features lean toward localization while deep features lean toward semantics. The intermediate layer (layer 18) offers the optimal balance—when semantics are ambiguous due to object symmetry or lack of texture, localization information ensures geometrically consistent matches.

Method

Overall Architecture

FoundPose adopts the classical "render-match-solve" paradigm, but replaces traditional handcrafted features with DINOv2 features:

1. Offline Modeling Phase: Render RGB-D templates of the 3D object model, extract DINOv2 patch descriptors, and register them to the 3D space.
2. Online Inference Phase: Crop query image \(\rightarrow\) bag-of-words template retrieval \(\rightarrow\) patch matching to establish 2D-3D correspondences \(\rightarrow\) PnP-RANSAC for pose solving \(\rightarrow\) featuremetric refinement.

Key Designs

1. Template Representation and 3D Registration

Given a textured 3D object model, render \(n\) RGB-D templates covering the \(SO(3)\) rotation group. For each template, extract \(14 \times 14\) non-overlapping patch descriptors \(\{\mathbf{p}_{t,i}\}_{i=1}^{m}\), and reduce their dimensionality to \(d\) via PCA:

\[\mathbf{p}_{t,i} = \phi_d(\mathbf{p}'_{t,i}), \quad \phi_d: \mathbb{R}^r \mapsto \mathbb{R}^d\]

Each patch descriptor is associated with a 3D position \(\mathbf{x}_j\) (projected from 2D to 3D using the depth channel and camera intrinsics), forming the template representation \(T_t = \{(\mathbf{p}_{t,j}, \mathbf{x}_j) | j \in M\}\).

Design Motivation: Registering DINOv2 descriptors to 3D space allows direct conversion of 2D patch matching into 2D-3D correspondences, avoiding the need for an additional depth estimation network.

2. Bag-of-Words Template Retrieval

To efficiently find the few templates most similar to the query from hundreds, DINOv2 patch descriptors are integrated into a classic bag-of-words framework:

• Perform \(k\)-means clustering on all template patch descriptors to obtain a visual vocabulary.
• Represent each template with a TF-IDF weighted bag-of-words vector: \(b_i = (n_{i,t}/n_t) \log(N/n_i)\).
• During inference, compute the bag-of-words vector of the cropped image and retrieve the \(h\) most similar templates using cosine similarity.

Key Advantages: - 15 times faster than MegaPose's coarse render-and-compare estimation. - Requires only 800 templates (vs MegaPose's 90K+), reducing memory footprint by 25 times. - Cosine similarity is robust to occlusions—visible parts of an occluded object still contribute active visual words. - Soft-assignment mitigates quantization errors.

3. Patch Matching Based on Intermediate DINOv2 Layers

For each retrieved template, match the patch descriptors of the query crop to the template descriptors via kNN search to establish 2D-3D correspondences \(C_t = \{(\mathbf{u}_i, \mathbf{x}_i)\}_{i=1}^{m}\).

Key Findings: Using descriptors from DINOv2 ViT-L's 18th layer (out of 23 layers) yields the best results. Moving from shallow to deep layers, features transition from being location-driven to being semantic-driven: - Layer 13: Different sides of the object show distinctly different colors (strong location information). - Layer 23: The entire object tends to have the same color (strong semantic information). - Layer 18: On symmetric/textureless objects, location information helps disambiguate semantically ambiguous correspondences.

4. Featuremetric Refinement

The coarse pose is estimated from 2D-3D correspondences via PnP-RANSAC. Due to the large patch size (\(14 \times 14\) px), a discrepancy exists between 2D centers and 3D projections. The pose is refined by minimizing the featuremetric error via Levenberg-Marquardt optimization:

\[(\mathbf{R}_r, \mathbf{t}_r) = \arg\min_{(\mathbf{R}, \mathbf{t})} \sum_{(\mathbf{p}_i, \mathbf{x}_i) \in T_t} \rho\left(\mathbf{p}_i - \mathbf{F}_q\left(\pi(\mathbf{R}\mathbf{x}_i + \mathbf{t})/s\right)\right)\]

where \(\rho\) is the Barron robust loss function, \(\mathbf{F}_q\) is the query feature map (bilinearly interpolated), and \(\pi\) is the 2D projection function.

Design Motivation: Similar to classic photometric alignment, but operating in the DINOv2 feature space, this aligns template patches to their optimal positions in the query image, effectively compensating for discretization errors caused by coarse sampling.

Loss & Training

FoundPose does not require any training—it completely relies on frozen DINOv2 weights. The offline modeling phase only requires: - Rendering 800 templates (approx. 25° angular interval). - Extracting patch descriptors and computing PCA (calculating top-256 components from valid patch descriptors of all templates). - Computing k-means clustering centroids (2048 visual words). - The entire modeling process takes < 5 minutes on a single GPU.

Key Experimental Results

Main Results

Comparison of Average Recall (AR) on 7 BOP benchmark datasets (without refinement):

Method	LM-O	T-LESS	TUD-L	IC-BIN	ITODD	HB	YCB-V	Average AR	Requires Training
FoundPose	39.6	33.8	46.7	23.9	20.4	50.8	45.2	37.2	✗
GigaPose	29.9	27.3	30.2	23.1	18.8	34.8	29.0	27.6	✓
GenFlow	25.0	21.5	30.0	16.8	15.4	28.3	27.7	23.5	✓
MegaPose	22.9	17.7	25.8	15.2	10.8	25.1	28.1	20.8	✓

With MegaPose refiner (5-hypothesis refinement):

Method	Average AR	Time (s)
FoundPose + Feat. + MegaPose	59.6	20.5
GigaPose + MegaPose	57.9	7.3
GenFlow + GenFlow	57.1	20.9
MegaPose + MegaPose	54.7	47.4

Ablation Study

Impact of different backbones/layers on coarse pose estimation accuracy:

Feature Extractor	LM-O	T-LESS	TUD-L	Average AR	Notes
DINOv2 ViT-L layer 18	39.6	33.8	46.7	37.2	Best intermediate layer
DINOv2 ViT-L layer 23	23.2	22.8	31.2	23.5	Last layer, overly semantic
DINOv2 ViT-S layer 9	34.0	31.6	42.7	34.0	Intermediate layer of a smaller model
SAM ViT-L layer 23	2.2	12.8	9.2	10.7	Unsuitable for pose estimation
Dense SIFT	3.2	2.6	6.5	7.6	Classical handcrafted features

Ablation on template retrieval methods:

Retrieval Method	Average AR	Notes
Bag-of-Words (Ours)	37.2	Robust, efficient
CLS token matching	18.2	Sensitive to occlusions
CLS token + black background	25.1	Improved but still insufficient
Direct top template pose	17.7	Verification of retrieval quality

Key Findings

1. DINOv2 intermediate-layer descriptors are key: Level 18 vs Level 23 brings a massive +13.7 AR boost, proving the importance of the localization-semantic balance.
2. Training-free outperforms training-based: FoundPose surpasses GigaPose (+9.6 AR), GenFlow (+13.7 AR), and MegaPose (+16.4 AR) without any training.
3. Featuremetric refinement is effective: Provides +5 AR (single-hypothesis) and is complementary to the MegaPose refiner.
4. Extremely high memory efficiency: Only 234MB per object (vs 5.6GB for OSOP/Nguyen, which is 25 times lower).
5. Segmentation quality is a bottleneck: Using GT masks yields a massive +6~19 point gain in AR.

Highlights & Insights

1. Paradigm Inspiration: Proves that the combination of a frozen foundation model + classic geometry can outperform end-to-end trained methods, providing strong evidence for "zero-shot, training-free deployment."
2. Value of Intermediate Features: Unlike the common practice of using the final layer's features, intermediate-layer features are found to be superior for pose estimation due to their blend of location awareness and semantic understanding.
3. Revival of Bag-of-Words: The classic text retrieval technique from 2003 is revitalized in 2024 by combining it with visual foundation models.
4. Synthetic-to-Real Generalization: Self-supervised training allows DINOv2's patch descriptors to be naturally consistent across domains without explicit domain adaptation.

Limitations & Future Work

1. FoundPose relies on segmentation masks provided by CNOS; the mask quality is the primary source of error.
2. It only supports RGB input and does not utilize depth information (making it not directly comparable to RGB-D methods like FoundationPose).
3. Patch granularity is coarse (\(14 \times 14\) px); precise localization depends on the refinement step.
4. Highly symmetric objects (such as cylinders) remain challenging.
5. The potential of more efficient foundation models (e.g., distilled models) to boost speed remains unexplored.

Related Work & Insights

• Feature layer analysis of DINOv2 can be transferred to other tasks requiring precise spatial correspondence (such as optical flow or depth estimation).
• The combined idea of Bag-of-Words + foundation model features can be used for other retrieval tasks.
• The featuremetric alignment framework can be extended to dynamic object tracking scenarios.

Rating

• Novelty: ⭐⭐⭐⭐ — Cleverly combines foundation model features with classic geometry; the discovery of intermediate-layer features is inspiring.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ — Comprehensive evaluation across 7 datasets, with rich ablation studies covering feature layers, retrieval methods, and various parameters.
• Writing Quality: ⭐⭐⭐⭐⭐ — Clear logic, well-motivated, and excellent visualizations (PCA visualization of features is very intuitive).
• Value: ⭐⭐⭐⭐⭐ — Training-free, low-memory, and highly accurate, offering exceptional value for real-world deployment.

Related Papers

• [ECCV 2024] GS-Pose: Category-Level Object Pose Estimation via Geometric and Semantic Correspondence
• [ICCV 2025] RayPose: Ray Bundling Diffusion for Template Views in Unseen 6D Object Pose Estimation
• [CVPR 2025] UNOPose: Unseen Object Pose Estimation with an Unposed RGB-D Reference Image
• [ICCV 2025] MixRI: Mixing Features of Reference Images for Novel Object Pose Estimation
• [ECCV 2024] U-COPE: Taking a Further Step to Universal 9D Category-Level Object Pose Estimation