Pose Anything Anywhere: Model-free Object Poses from Arbitrary References¶

Conference: ECCV2026
arXiv: 2606.23634
Code: None
Area: Robotics / 3D Vision
Keywords: 6D Pose Estimation, Model-free Pose Estimation, Multi-view Geometry, Cross-view Feature Matching, Robot Perception

TL;DR¶

PANY proposes a unified model-free 6D pose estimation framework that jointly trains multi-view geometric reasoning and cross-view correspondence learning in an end-to-end manner. It can directly estimate the 6D pose of unseen objects from arbitrary numbers of sparse reference images (including optional pose-free auxiliary views) without requiring CAD models or pre-registration, outperforming prior methods by a large margin on benchmarks such as YCB-V and LM-O.

Background & Motivation¶

Open-world robotics requires perceiving unseen objects and performing manipulation tasks like grasping and placing without prior knowledge, and the key prerequisite is estimating the 6D pose of objects. Existing methods roughly fall into three categories: model-based pipelines (e.g., FoundPose, RayPose) rely on precise CAD templates to generate rendered images for pose matching, which achieves high accuracy but demands acquiring CAD models for every new object, making large-scale deployment extremely costly; "reconstruction-first" pipelines (e.g., OnePose++, FoundationPose) employ multi-view reconstruction followed by model-based pose solvers, but require dense, pose-annotated image sequences and take a long time to register, making them unsuitable for quick onboarding of new objects; pairwise model-free matching methods (e.g., One2Any, SingRef6D) only employ a single reference image to establish dense correspondences or regress poses from the query, relying heavily on single-view information. When severe occlusion or large viewpoint changes occur, the visual overlap between the reference and the query is extremely low, making the problem inherently ill-posed—a single image fundamentally cannot cover the occluded side of the object.

The key challenge of this dilemma lies in: while actual users can most easily provide a small number of casual photos (without pose information, sometimes just one or two), current methods face insufficient geometric information from a single reference image to resolve pose ambiguity under large viewpoint variations and heavy occlusions. While multi-view constraints can clearly reduce uncertainty, traditional multi-view geometry pipelines require dense sequences with known poses and explicit 3D reconstruction, which is too demanding for open-world scenarios. This paper's key insight is to enable the model to jointly reason about geometric structures and cross-view correspondences from multi-view inputs during the training phase. Consequently, during inference, the model can directly estimate poses from a single reference image, while also flexibly utilizing an arbitrary number of pose-free "auxiliary views" that act as intermediate bridges to bridge the wide-baseline gap between the query and anchor views. Core Idea: Jointly train a multi-view geometric reasoning backbone and cross-view 3D correspondence learning in an end-to-end framework, enabling the model to support both direct matching with a single reference and gradual alignment to the anchor coordinate system using pose-free auxiliary views via pose graph canonical registration during inference, thereby significantly improving pose robustness under occlusion and large viewpoint variations.

Method¶

Overall Architecture¶

The input to PANY consists of three parts: a query image, an anchor reference image that defines the canonical pose of the object, and zero or more auxiliary views without any pose annotations. All views are first tokenized by a shared DINOv2 encoder and sent to a VGGT-based multi-view Transformer backbone. This backbone alternately performs intra-frame self-attention (capturing local appearance within each view) and cross-view cross-attention (establishing geometric associations across all views) to output visual embeddings for each view, which are then upsampled by a DPT head to predict depth maps and 3D point maps. Meanwhile, a cross-view 3D matching head fuses visual features and geometric features into rotation-invariant 3D descriptors, building dense cross-view 3D correspondences via local contrastive learning. During inference, if the visual overlap between the query and the anchor is insufficient, the auxiliary views gradually align and register to the anchor coordinate system via a pose graph registration process; if the overlap is sufficient, direct query-to-anchor point map alignment is performed. Finally, Kabsch-Umeyama (for RGB-D input) or PnP (for RGB-only input) is used to recover the 6D pose of the query.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query + Anchor<br/>+ Auxiliary Views (M views)"] --> B["DINOv2 Encoder<br/>Patching + Tokenization"]
    B --> C["Multi-View Transformer Backbone<br/>Intra-frame Self-attention + Cross-view Cross-attention<br/>Based on VGGT + LoRA Fine-tuning"]
    C --> D["DPT Decoding Head<br/>Depth + 3D Point Map Prediction"]
    C --> E["Cross-View 3D Matching Head<br/>Visual-Geometric Feature Fusion -> Rotation-invariant Descriptors"]
    D --> F{"Auxiliary Views (M ≥ 1)?"}
    E --> F
    F -->|No| G["Direct Query-Anchor<br/>Point Map Kabsch Alignment"]
    F -->|Yes| H["Pose Graph Construction<br/>Valid Correspondences ≥ η=500<br/>RANSAC + Umeyama"]
    H --> I["Traverse Pose Graph Paths<br/>Step-by-step Transformation Compositions -> Anchor Frame<br/>Canonical Registration"]
    I --> J["Kabsch-Umeyama (PnP)<br/>Recover 6D Query Pose"]
    G --> J
    J --> K["6D Pose Output"]

Key Designs¶

1. Multi-View Geometric Reasoning Backbone: Adapting from Scene-Level Geometric Foundation Models to Object-Centric Canonical Alignment

PANY adopts VGGT as the geometric foundation model, which natively possesses the capability to jointly predict depth maps, point maps, and camera parameters from multiple pose-free RGB images. However, VGGT is trained under scene-level tracking-style supervision, meaning its output representations exhibit viewpoint dependency—the same 3D point falls under different relative coordinate systems in two views with large viewpoint differences. This is a fatal flaw for object-level pose estimation that requires consistent cross-view alignment. PANY's improvement strategy is to freeze the VGGT backbone and only fine-tune it via LoRA (rank=8, alpha=32), while introducing a cross-view matching head as an additional supervisory signal. During training, each sample randomly selects 2-8 reference views (for a total of 3-9 input views) to cover various viewpoint combinations. Contrastive learning in the matching head forces the descriptors of corresponding points from different viewpoints to be close in a shared 3D space, which gradually eliminates viewpoint dependency through joint optimization of the backbone. Consequently, the predicted point maps are naturally aligned under a shared coordinate system. This training strategy enables the model to handle an arbitrary number of view inputs, laying a foundation for flexible adaptation during inference.

2. Cross-View 3D Matching and Geometric-Aware Feature Fusion: Establishing Robust Correspondences across Wide Baselines with Local Contrastive Learning

Relying solely on predicted point maps is insufficient to handle dense matching under large viewpoint variations. PANY designs a cross-view 3D matching module: first, rotation-invariant geometric features are extracted from the point maps of each view using a geometric encoder (utilizing rotation-invariant positional embeddings). Meanwhile, pixel-level features from the visual backbone are lifted to the 3D space via coordinate indexing to align the two modalities, which are then fused into geometric-aware tokens. These tokens are fed into a geometric Transformer that interleaves sparse geometric attention and dense linear attention to output rotation-invariant 3D descriptors. During training, the attention matrix \(A_{ij}=F_i^m(F_j^m)^\top\) is computed for all valid view pairs, and a bidirectional cross-entropy loss \(\mathcal{L}_{ij}=\mathrm{CE}(A_{ij},\hat{Y}_i)+\mathrm{CE}(A_{ij}^\top,\hat{Y}_j)\) is used to supervise the correspondences. The correspondence labels are defined by a nearest-neighbor distance threshold (\(\delta_\text{dis}\)) in the shared coordinate system, where only points within this distance have valid correspondences. The elegance of this design lies in performing matching within the predicted shared 3D point map space rather than on the 2D image plane, naturally avoiding projection distortions caused by viewpoint variations, while the rotation-invariant features keep matching robust against changes in texture and illumination.

3. Pose Graph Canonical Registration: Pose-Free Auxiliary Views Acting as Wide-Baseline Bridges

This represents the key mechanism of PANY during inference. When the visual overlap between the query and the anchor is extremely small (e.g., the front and back of an object), direct matching is virtually impossible. PANY utilizes user-provided, pose-free auxiliary views as intermediate bridges to transfer geometric information: after all views generate predicted point maps and matching descriptors through the backbone, a pose graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) is constructed with views/point clouds as nodes and edges defined by valid correspondences exceeding a threshold (\(\eta=500\)). For each edge, a similarity transformation \(\mathbf{T}_{i\leftarrow j}\) is robustly estimated using RANSAC + Umeyama. Starting from the anchor node, the transformations are composed along the pose graph paths, \(\mathbf{T}_{i\rightarrow A}=\prod_{(k,l)\in\pi(i,A)}\mathbf{T}_{k\leftarrow l}\), to register the point clouds of all auxiliary views into the anchor coordinate system as \(P̃_i=\mathbf{T}_{i\rightarrow A}\hat{P}_i\). This allows reliable recovery of the query pose, even in the complete absence of a single reference view that sufficiently overlaps with the query, by passing geometric information step-by-step through multi-view "relays". Experimental results show that performance basically saturates with just 4 auxiliary views, indicating that incremental geometric coverage indeed eliminates viewpoint uncertainty effectively.

Loss & Training¶

The training objective is jointly optimized by three components: a depth regression L1 loss \(\mathcal{L}_\text{depth}\), a point map regression L1 loss \(\mathcal{L}_\text{point}\), and a cross-view matching bidirectional cross-entropy loss \(\mathcal{L}_\text{match}\), resulting in the total loss \(\mathcal{L}=\lambda_d\mathcal{L}_\text{depth}+\lambda_p\mathcal{L}_\text{point}+\lambda_m\mathcal{L}_\text{match}\), with weights set to \(\lambda_d=1.0, \lambda_p=1.0, \lambda_m=0.05\). The weight of the matching loss is lower because its gradient signal derives from the contrastive nature of the attention matrix rather than regression pressure. The model is trained on the synthetic Omni6DPose dataset (5000+ CAD objects, 149 categories, 2 million+ RGB-D images) for 15 epochs, with 500 iterations per epoch and a batch size of 16, taking approximately 36 hours on two A100 GPUs.

Key Experimental Results¶

Main Results¶

Dataset	Modality	Configuration	Metric	PANY	Prev. SOTA	Gain
YCB-V	RGB	Single Ref	ADD AUC	92.5%	84.4% (One2Any)	+8.1%
YCB-V	RGB	1 Anchor + 8 Aux	ADD AUC	94.6%	84.4% (One2Any)	+10.2%
LINEMOD	RGB	Single Ref	ADD-0.1d	55.3%	52.6% (One2Any)	+2.7%
LINEMOD	RGB	1 Anchor + 8 Aux	ADD-0.1d	87.4%	52.6% (One2Any)	+34.8%
Real275	RGB-D	Single Ref	AR	81.8%	54.9% (One2Any)	+26.9%
Real275	RGB-D	Single Ref	ADD-0.1d	89.3%	53.5% (Any6D)	+35.8%
Toyota-Light	RGB-D	Single Ref	AR	51.1%	43.3% (Any6D)	+7.8%
LM-O (CNOS mask)	RGB	1 Anchor + 8 Aux	AR	42.3%	39.7% (FoundPose* 8 Anchors)	+2.6%

Note: FoundPose is replicated under the same conditions as in the literature, using 8 posed anchor views; PANY's auxiliary views are unposed.

Ablation Study¶

Design Ablation (Real275, single anchor without auxiliary views):

Configuration	AR	ADD-0.1d	Description
Geometry Only (No matching head)	56.6%	23.6%	Direct Kabsch alignment on predicted point maps
+ Visual Matching (No geometric fusion)	57.6%	25.4%	Visual embedding only for matching
Full Model (Matching + Geometric fusion)	59.3%	27.8%	Geometry-aware feature fusion

Backbone Ablation (same setting):

Configuration	AR	ADD-0.1d	Description
pure DUSt3R	27.8%	16.8%	Replacing backbone + same solver
pure MASt3R	29.2%	17.6%	Ditto
pure VGGT	43.6%	33.4%	Ditto
Ours Full	51.1%	39.8%	Full system

Key Findings¶

Matching Head + Geometric Feature Fusion contributes the most: Compared to the geometry-only baseline, the full model's ADD-0.1d improves from 23.6% to 27.8% (Table 4b). Furthermore, comparing the backbone-only VGGT baseline (33.4% vs. full model's 39.8%, Table 4a) demonstrates that the performance gain stems from the matching head design rather than mere backbone replacement.
Performance tends to saturate at 4 auxiliary views, and the default of 8 views is sufficient; the choice of anchor view has minimal impact on final results, indicating that the method is robust to anchors.
On LM-O (CNOS mask), PANY with just 1 anchor + 8 pose-free auxiliary views outperforms model-based methods using dense CAD templates (FoundPose 39.7% vs. PANY 42.3%), proving the competitiveness of model-free solutions under sparse references.

Highlights & Insights¶

Viewpoint-Decoupled Geometric Features: Fusing visual features into the predicted 3D point map space and enforcing viewpoint consistency via contrastive learning avoids matching degradation under large viewpoint variations in 2D features—this is the core reason PANY significantly outperforms One2Any under the single reference setting.
The "Bridge" Paradigm of Auxiliary Views: It does not require any pose annotations for auxiliary views, acting merely as a conduit for geometric information. This contrasts sharply with traditional methods demanding dense posed sequences, making it much closer to real-world user habits.
Elastic Train-Inference Design: Using 3-9 random views during training and flexibly adapting to any quantity (1+M views) during inference. This strategy teaches the model to handle both information redundancy and sparsity under extreme conditions.
Pose Graph Registration converts the N-view alignment problem into a composition of pairwise view-to-view transformations, employing RANSAC to filter out wrong correspondences and Umeyama to estimate rigid transformations, showing good scalability while maintaining accuracy.

Limitations & Future Work¶

Failure cases may still occur under extreme occlusions or when the query-reference overlap is minimal (e.g., when the object only shows a single edge), where even multi-view relays cannot successfully establish valid correspondences.
Pose ambiguity for symmetric objects (such as featureless cylinders) remains unresolved: geometric cues are insufficient to determine the orientation, requiring the introduction of semantic priors or symmetry assumptions.
Currently, the model relies on external detectors like CNOS to provide segmentation masks; extending this to an end-to-end joint detection and localization framework is a natural direction.
Fine-tuning VGGT using LoRA might limit the generalization upper bound in new domains; future work could explore larger parameter spaces or adapter-free architectures.

vs One2Any: One2Any predicts RGB-D matching using a single reference via ROC (Reference Object Coordinate) maps, which is competitive in single-reference scenarios. PANY's multi-view reasoning enables it to reach nearly twice the accuracy under occlusion and large viewpoint variations (e.g., YCB-V ADD AUC from 84.4% to 92.5%).
vs FoundPose/RayPose: These are model-based methods relying on CAD templates or rendering comparison, whereas PANY completely frees itself from CAD models, making it suitable for zero-preparation novel objects.
vs VGGT/DUSt3R: Point maps output by scene-level geometric foundation models are viewpoint-dependent; PANY adapts them to object-centric canonical alignment through cross-view contrastive learning.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ For the first time, a multi-view geometric reasoning framework is unified with cross-view correspondence learning for model-free pose estimation. With no need for CADs or dense pose-annotated sequences, it supports both single reference and sparse multi-reference inputs, making it a rare, all-in-one milestone solution in this field.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across 5 real-world benchmarks, covering multiple configurations like single/multi-reference, RGB/RGB-D, and CNOS/GT masks, with comprehensive and solid ablation studies.
Writing Quality: ⭐⭐⭐☆☆ While the technical content is comprehensive and the method section is clearly structured, the introduction and motivation sections are somewhat lengthy and could be more concise.
Value: ⭐⭐⭐⭐⭐ Significantly lowers the preparation threshold for pose estimation in open-world robot deployment, showing a highly promising prospect for practical applications.