Skip to content

Gravity-aware partially calibrated absolute pose estimation from affine- or rotation-covariant features

Conference: ECCV 2026
Paper: ECCV 2026 Paper
Area: 3D Vision
Keywords: absolute pose estimation, camera calibration, gravity prior, affine correspondence, minimal solver

TL;DR

Addressing absolute pose estimation with unknown focal length (PnPf), this paper incorporates IMU gravity priors and local geometric attributes of feature descriptors (affine or rotation-covariant) to derive novel polynomial constraints and construct two efficient solvers—UP1PfAC from a single affine correspondence and UP2PfORI from two orientation-covariant features—slashing sample requirements to 1 or 2 features while drastically accelerating RANSAC robust localization and focal length recovery.

Background & Motivation

In emerging applications such as extended reality (XR), autonomous drones, and collaborative multi-user mixed reality, client devices must achieve rapid and precise global relocalization against a pre-built 3D scene map. At the heart of visual localization pipelines lies the Perspective-n-Point (PnP) problem. However, in consumer-grade devices like smartphones and wearable headsets, the camera focal length is often unknown or dynamically changing due to continuous autofocus. Consequently, systems must jointly estimate camera absolute pose and focal length (partially calibrated absolute pose estimation, PnPf). The standard point-based PnPf formulation possesses seven degrees of freedom and demands at least 3.5 to 4 point correspondences; in real-world scenarios plagued by high outlier ratios, the combinatorial sampling overhead required by RANSAC scales exponentially with the minimal sample size, creating a severe bottleneck for real-time latency and energy consumption.

Meanwhile, contemporary mobile platforms routinely integrate high-precision inertial measurement units (IMUs) capable of delivering reliable gravity vectors with minimal delay. This gravity alignment eliminates camera pitch and roll, confining the unknown rotational ambiguity strictly to a single yaw angle about the vertical axis. Prior gravity-aligned absolute pose solvers (e.g., UP2P, UP1SIFT) assume camera intrinsic parameters are fully known beforehand, while existing point-based semi-calibrated solvers (e.g., UP2.5Pf) rely exclusively on 2D point positions, overlooking the rich local differential geometry (scale, rotation, and affine covariance) inherent in descriptors such as SIFT, ORB, or learned features like AffNet and S3Esti. How to gracefully fuse IMU gravity measurements with local feature geometry to recover unknown focal length has remained an unaddressed challenge in geometric computer vision.

Introducing affine features to partially calibrated absolute pose estimation introduces a major technical barrier: the unknown focal length nonlinearly couples with local affine transformations and point projection equations, inducing dramatic monomial expansion that renders classical algebraic elimination intractable for compact, closed-form polynomial solvers. This paper attacks the problem by leveraging gravity alignment to decouple rotation, parameterizing yaw and focal length directly within the differential affine constraints, and uncovering a low-rank nullspace structure in the augmented translation matrix. Core idea: exploit IMU gravity priors to reduce rotation to a single yaw parameter, incorporate unknown focal length into local affine and orientation-covariant geometric equations, and construct quartic closed-form solvers requiring only a single affine correspondence (UP1PfAC) or two orientation-covariant features (UP2PfORI) with zero-redundancy hypothesis selection.

Method

Overall Architecture

The proposed joint pose and focal length estimation pipeline operates on correspondences between a query image and a 3D point cloud map. The input consists of a query image with unknown focal length \(f\), an IMU-derived gravity vector, and reference image 2D keypoints associated with 3D map points and estimated surface normal vectors. Depending on the feature detector capabilities, the pipeline branches into one of two minimal polynomial solvers: if the features provide full local affine transformations (e.g., SIFT or AffNet), the UP1PfAC solver is executed; if only rotation orientation covariance is available (e.g., ORB or SuperPoint with orientation estimation), the UP2PfORI solver is invoked.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query Image + IMU Gravity Prior + 3D-2D Matches"] --> B["Gravity Alignment & Parameterization<br/>Align pitch/roll, set yaw r=tan(θ/2), normalize focal length"]
    B --> C{"Feature Geometry Branch"}
    C -->|Single Affine Correspondence| D["UP1PfAC: 5x4 Matrix Nullspace Elimination<br/>Subdeterminant factorization yields quartic polynomial in r"]
    C -->|Two Orientation-Covariant Features| E["UP2PfORI: Translation Elimination & Orientation Coupling<br/>Form 2x2 polynomial system and eliminate to solve quartic r"]
    D --> F["Backsubstitution for Focal Length f & Translation t"]
    E --> F
    F --> G["Algebraic Residual Verification: Single-Solution Selection<br/>Filter algebraic spurious roots via unused constraint"]
    G --> H["Output: High-Precision Pose (R, t) & Focal Length f"]

In the overall pipeline, the camera rotation is decomposed into a known alignment matrix \(R_{xz}\) and an unknown rotation \(R_y(\theta)\) around the gravity vertical axis, parameterized via the tangent half-angle \(r = \tan(\theta/2)\). A structured linear matrix equation in the translation vector is formulated, from which translation is eliminated using nullspace determinant conditions. This yields an analytical quartic equation in \(r\), solvable in closed form. Finally, algebraically redundant residual equations are evaluated to discard extraneous roots and pass a single clean hypothesis directly to GC-RANSAC.

Key Designs

1. Gravity-aligned affine constraint formulation with unknown focal length: preventing monomial explosion

In the world coordinate frame, the local affine transformation matrix \(A\) between query and reference images depends on relative rotation \(R\), relative translation \(t\), the 3D surface normal \(n_{\text{ref}}\), and reference depth \(d\). When the query focal length \(f\) is unknown, normalized image coordinates transform as \(p_{\text{query}} \mapsto p_{\text{query}}/f\), while the affine transformation scales as \(A \mapsto (f_{\text{ref}}/f) A\). To circumvent high-degree nonlinear coupling between \(r\), \(f\), and translation components, the four affine components are expanded and combined with point projection constraints. Multiplying both sides by the homogenization factor \(m f\) (where \(m\) contains the projective depth term) preserves a strictly linear dependence on the query camera translation vector \(t_{\text{query}}\):

\[\begin{bmatrix} a_1 / f \\ a_2 / f \\ a_3 / f \\ a_4 / f \end{bmatrix} = \frac{d}{m} \begin{bmatrix} (b r_{11} - p'_{\text{ref}, 1} n_{\text{ref}, 1}) - \frac{p_{\text{query}, 1}}{f} (b r_{31} - p'_{\text{ref}, 3} n_{\text{ref}, 1}) \\ (b r_{12} - p'_{\text{ref}, 1} n_{\text{ref}, 2}) - \frac{p_{\text{query}, 1}}{f} (b r_{32} - p'_{\text{ref}, 3} n_{\text{ref}, 2}) \\ (b r_{21} - p'_{\text{ref}, 2} n_{\text{ref}, 1}) - \frac{p_{\text{query}, 2}}{f} (b r_{31} - p'_{\text{ref}, 3} n_{\text{ref}, 1}) \\ (b r_{22} - p'_{\text{ref}, 2} n_{\text{ref}, 2}) - \frac{p_{\text{query}, 2}}{f} (b r_{32} - p'_{\text{ref}, 3} n_{\text{ref}, 2}) \end{bmatrix}\]

This compact formulation confines the unknown focal length to structured matrix coefficients, setting the foundation for closed-form elimination.

2. UP1PfAC minimal solver: 4th-degree algebraic reduction and nullspace elimination from one affine match

A single affine correspondence yields 4 affine equations and 2 point projection equations, totaling 6 equations for 5 unknowns (\(r\), \(t_{\text{query}} \in \mathbb{R}^3\), and \(f\)). To construct a minimal solver, the 4th affine constraint is temporarily set aside, gathering the remaining 5 equations into a homogeneous system \(M(r, f) [t_{\text{query}}^\top, 1]^\top = \mathbf{0}\), where \(M(r, f)\) has dimension \(5 \times 4\). For non-trivial solutions to exist, all \(4 \times 4\) subdeterminants of \(M\) must vanish. By introducing the substitution \(\xi = f_{\text{ref}} f (1 + r^2)\), the non-trivial subdeterminants factor out the non-zero trivial term \((1 + r^2)^2 f^2\), leaving equations strictly linear in \(f\):

\[(1 + r^2)^2 \bar{M}(r) \begin{bmatrix} f \\ 1 \end{bmatrix} = \mathbf{0}\]

Here \(\bar{M}(r)\) is a \(2 \times 2\) polynomial matrix. Enforcing \(\det(\bar{M}(r)) = 0\) yields a quartic polynomial \(\bar{h}(r) = 0\) in the single variable \(r\). Because the polynomial is of degree 4, all roots are solved analytically via Ferrari's formula in nanoseconds. Backsubstitution into the \(2 \times 2\) system directly isolates \(f\), followed by linear recovery of \(t_{\text{query}}\).

3. UP2PfORI minimal solver: two orientation-covariant features across multiple reference views

When full affine shapes are unavailable and only local keypoint orientations are provided (as in ORB or SuperPoint), two correspondences provide 4 point projection equations and 2 scale-free orientation constraints that are completely independent of translation. Assembling the 4 projection equations yields a \(4 \times 4\) linear system in translation with determinant condition \(\det(M) = 0\). Factoring out \((1 + r^2)^2 f^2\) produces a bivariate polynomial \(\bar{g}(r, f) = 0\). Pairing \(\bar{g}(r, f)\) with one orientation constraint produces another \(2 \times 2\) linear system in \(f\):

\[\bar{M}_{\text{ori}}(r) \begin{bmatrix} f \\ 1 \end{bmatrix} = \mathbf{0}\]

Setting \(\det(\bar{M}_{\text{ori}}(r)) = 0\) once again generates a single univariate quartic equation in \(r\), guaranteeing at most 4 analytical roots. Crucially, because each point projection is independently parameterized, UP2PfORI naturally accepts features originating from distinct reference frames, substantially broadening the valid sampling pool in large-scale visual localization.

4. Algebraic residual verification: single-hypothesis filtering

A 4th-degree polynomial yields up to 4 real candidate solutions. Standard RANSAC pipelines evaluate consensus across all generated roots, multiplying inlier counting overhead. Both proposed solvers solve this by reintroducing the discarded algebraic constraints: UP1PfAC evaluates the residual of the omitted 4th affine equation, while UP2PfORI checks the unused 2nd orientation constraint. By evaluating this single residual across the candidate roots, the solver automatically identifies the true physical configuration and outputs exactly one hypothesis to RANSAC, minimizing verification overhead.

Key Experimental Results

Main Results

The solvers were comprehensively evaluated within Graph-Cut RANSAC (GC-RANSAC) on two real-world localization benchmarks: Cambridge Landmarks and Aachen Day-Night v1.1. On Cambridge Landmarks, evaluations were conducted using both learned features (SuperPoint + LightGlue + S3Esti) and handcrafted features (RootSIFT + Nearest Neighbors), measuring median translation, rotation, and focal length errors alongside runtime.

Feature Setup Solver GreatCourt (cm / ° / ferr / ms) KingsCollege (cm / ° / ferr / ms) OldHospital (cm / ° / ferr / ms) ShopFacade (cm / ° / ferr / ms) StMarysChurch (cm / ° / ferr / ms)
SP+LG+S3Esti P4Pf (Point baseline) 61.5 / 0.14° / 0.010 / 26.9 36.3 / 0.30° / 0.011 / 48.3 58.7 / 0.42° / 0.015 / 40.9 14.6 / 0.30° / 0.011 / 39.9 23.9 / 0.32° / 0.014 / 42.3
SP+LG+S3Esti P3.5Pf (Point baseline) 61.3 / 0.14° / 0.010 / 28.0 36.0 / 0.30° / 0.011 / 45.2 58.0 / 0.42° / 0.015 / 38.6 14.6 / 0.30° / 0.011 / 41.4 23.9 / 0.32° / 0.014 / 41.2
SP+LG+S3Esti UP2.5Pf (Point + gravity) 61.6 / 0.14° / 0.010 / 23.8 35.9 / 0.29° / 0.011 / 41.1 57.2 / 0.42° / 0.015 / 38.0 14.4 / 0.30° / 0.011 / 35.4 23.7 / 0.32° / 0.013 / 37.0
SP+LG+S3Esti UP1PfAC (Ours) 57.7 / 0.13° / 0.009 / 22.6 35.6 / 0.28° / 0.012 / 39.9 52.3 / 0.38° / 0.013 / 34.2 10.6 / 0.26° / 0.008 / 33.9 22.6 / 0.31° / 0.014 / 34.3
SP+LG+S3Esti UP2PfORI (Ours) 57.9 / 0.13° / 0.009 / 24.0 35.3 / 0.28° / 0.012 / 41.7 52.2 / 0.37° / 0.014 / 37.1 10.9 / 0.26° / 0.008 / 34.3 22.9 / 0.31° / 0.014 / 38.2
SIFT+NN P4Pf (Point baseline) 75.8 / 0.16° / 0.011 / 23.3 37.9 / 0.31° / 0.012 / 41.9 51.1 / 0.39° / 0.017 / 25.9 11.2 / 0.28° / 0.008 / 23.6 28.7 / 0.37° / 0.018 / 28.4
SIFT+NN UP2.5Pf (Point + gravity) 75.6 / 0.16° / 0.011 / 20.5 37.8 / 0.31° / 0.012 / 40.1 51.9 / 0.39° / 0.017 / 24.2 11.2 / 0.28° / 0.008 / 20.6 28.4 / 0.38° / 0.018 / 26.4
SIFT+NN UP1PfAC (Ours) 70.5 / 0.15° / 0.012 / 16.7 37.6 / 0.30° / 0.012 / 35.5 48.4 / 0.33° / 0.018 / 21.0 11.1 / 0.27° / 0.008 / 20.8 27.7 / 0.38° / 0.017 / 24.2
SIFT+NN UP2PfORI (Ours) 70.8 / 0.15° / 0.011 / 17.7 37.8 / 0.30° / 0.012 / 35.4 50.4 / 0.36° / 0.018 / 23.8 11.0 / 0.27° / 0.008 / 21.6 28.1 / 0.38° / 0.018 / 27.1

On the challenging Aachen Day-Night v1.1 benchmark (characterized by day-night illumination domain shifts and varied mobile capture devices), the proposed solvers delivered superior localization recall across all thresholds:

Feature Pipeline Solver Day Recall (0.25m/2°, 0.5m/5°, 5m/10°) Night Recall (0.25m/2°, 0.5m/5°, 5m/10°) Focal Error (ferr) Time (ms)
SP+LG+S3Esti P4Pf 46.1% / 70.0% / 96.8% 54.5% / 74.9% / 95.8% 0.010 16.7
SP+LG+S3Esti P3.5Pf 45.6% / 69.7% / 96.7% 54.5% / 74.9% / 95.3% 0.010 20.3
SP+LG+S3Esti UP2.5Pf 46.5% / 68.8% / 96.6% 55.0% / 73.8% / 95.3% 0.010 18.6
SP+LG+S3Esti UP1PfAC (Ours) 48.5% / 70.8% / 97.9% 60.2% / 80.1% / 97.4% 0.009 16.7
SP+LG+S3Esti UP2PfORI (Ours) 48.4% / 71.2% / 98.2% 62.3% / 80.1% / 97.4% 0.009 19.8
SIFT+NN P4Pf 37.4% / 57.0% / 85.9% 19.4% / 20.9% / 26.2% 0.019 37.9
SIFT+NN UP2.5Pf 37.9% / 57.4% / 85.4% 18.8% / 21.5% / 26.2% 0.018 32.2
SIFT+NN UP1PfAC (Ours) 38.8% / 58.6% / 86.3% 18.3% / 22.0% / 27.2% 0.017 28.0
SIFT+NN UP2PfORI (Ours) 39.7% / 58.4% / 86.3% 19.4% / 22.5% / 26.7% 0.017 30.8

Ablation Study & Empirical Analysis

Synthetic benchmarking evaluated standalone solver execution speed (single call), numerical stability over 5,000 noise-free trials, and convergence speed in RANSAC under heavy outlier contamination:

Solver Minimal Sample Type Unknown Focal Length Single-Call Time (ns) Noise-Free Rotation Error Noise-Free Focal Error Time to 400 Inliers (50% Outliers)
P3.5Pf 3.5 points Yes 19,118 ns \(< 10^{-13}\) \(< 10^{-13}\) \(\sim 0.85\) ms
P4Pf 4 points Yes 3,179 ns \(< 10^{-14}\) \(< 10^{-14}\) \(\sim 0.60\) ms
UP2.5Pf 2.5 points + gravity Yes 642 ns \(< 10^{-14}\) \(< 10^{-14}\) \(\sim 0.35\) ms
UP1PfAC (Ours) 1 affine match + gravity Yes 2,586 ns \(< 10^{-14}\) \(< 10^{-14}\) \(\sim 0.15\) ms
UP2PfORI (Ours) 2 oriented features + gravity Yes 2,149 ns \(< 10^{-14}\) \(< 10^{-14}\) \(\sim 0.22\) ms
UP2P (Baseline) 2 points + gravity No (known) 484 ns \(< 10^{-15}\) - -
UP1SIFT (Baseline) 1 SIFT + gravity No (known) 1,448 ns \(< 10^{-14}\) - -
P1AC (Baseline) 1 affine match No (known) 2,740 ns \(< 10^{-13}\) - -

Key Findings

  • Sample size dictates end-to-end RANSAC runtime: While UP2.5Pf is the fastest single-call solver (642 ns), UP1PfAC requires only a single correspondence to instantiate a hypothesis. At a 50% outlier ratio, UP1PfAC converges to the consensus set over \(2\times\) faster than UP2.5Pf and \(4\times\) to \(6\times\) faster than point-based solvers (P4Pf/P3.5Pf). Under 70% outlier ratios, 4-point methods experience exponential latency spikes, whereas 1-point and 2-point solvers maintain virtually flat response times.
  • Robustness against composite noise: When subjected to 1.2 px image point noise, up to 1.0° surface normal errors, and 0.2° IMU gravity tilt, both UP1PfAC and UP2PfORI achieve normalized focal length errors below 0.9% to 1.4%, visibly outperforming 4-point baselines (1.5% to 1.9%).
  • Geometric prior vs. computational burden trade-off: UP1PfAC provides the strongest geometric constraints and lowest overall positioning error. Conversely, UP2PfORI avoids affine neighborhood fitting, uses simpler keypoint orientation cues (compatible with binary ORB), and allows cross-image reference matching, providing an appealing alternative when affine extraction is unavailable.

Highlights & Insights

  • First integration of differential affine constraints and gravity into partially calibrated PnP: The work successfully bridges the gap between local feature geometry and IMU inertial priors for unknown focal length absolute pose estimation, turning what was once an intractable algebraic problem into an exact quartic polynomial.
  • Zero-space algebraic reduction on augmented translation matrix: By structuring the 5-unknown problem into an augmented linear translation equation and applying subdeterminant vanishing conditions, the method eliminates translation without Groebner basis machinery, arriving directly at a closed-form Ferrari quartic.
  • Single-hypothesis output via residual evaluation: Leveraging discarded redundant constraints at solver-level to prune extraneous roots guarantees that only one optimal hypothesis enters the consensus loop, eliminating wasted inlier counts in modern RANSAC frameworks.

Limitations & Future Work

  • Reliance on local planar geometry and surface normals: UP1PfAC assumes local planarity and requires 3D surface normal estimates from the map. In sparse SfM models with noisy normals or near high-curvature geometric edges, affine constraints can degrade.
  • Focal length and translation scale ambiguity: Partially calibrated methods exhibit slightly higher translation errors compared to fully calibrated solvers (e.g., P1AC/UP2P) due to the inherent coupling between focal length and camera distance along the optical axis.
  • Future directions: Integrating dense geometric normals from modern neural implicit representations (e.g., NeRF or 3D Gaussian Splatting), and deploying the solver directly onto embedded low-power XR headsets for real-time visual-inertial relocalization.
  • vs P4Pf / P3.5Pf (Kukelova et al. / Larsson et al.): Prior semi-calibrated solvers rely on 3.5 to 4 points without IMU acceleration. The proposed method incorporates gravity alignment and affine constraints, dropping the minimal sample size to 1 and speeding up robust consensus by an order of magnitude under outliers.
  • vs UP2.5Pf (Kukelova et al.): UP2.5Pf exploits gravity but remains bound to 2.5 points without utilizing descriptor geometry. UP1PfAC solves the problem from a single visual landmark and introduces an internal algebraic pruning step.
  • vs UP1SIFT / P1AC (Ventura et al.): UP1SIFT and P1AC enforce fixed, known intrinsic calibration. The proposed solvers treat focal length as an unknown algebraic variable, accommodating autofocus and uncalibrated consumer hardware.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneered polynomial solvers combining IMU gravity priors and affine/orientation features for semi-calibrated absolute pose estimation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive synthetic noise ablations, runtime breakdowns, and rigorous benchmarks on Cambridge Landmarks and Aachen Day-Night]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary mathematical derivations, elegant matrix factorizations, and transparent analysis of algebraic nullspaces]
  • Value: ⭐⭐⭐⭐⭐ [Provides an essential low-latency building block for mobile XR, robotics, and uncalibrated visual localization systems]