Rolling Shutter Relative Pose Estimation Made Practical

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/danini/rolling_shutter_made_practical
Area: 3D Vision
Keywords: Rolling Shutter Geometry, Relative Pose Estimation, Affine Correspondences, Epipolar Geometry, Minimal Algebraic Solver

TL;DR

This paper introduces affine correspondences (ACs) into rolling shutter (RS) relative pose estimation for the first time, deriving RS-corrected affine constraints that account for point perturbation induced scanline shifts, and builds a linearized algebraic solver using 7 ACs with algebraic degree 20 that reduces RANSAC iterations by orders of magnitude while uniquely resolving the severe translational velocity coupling.

Background & Motivation

Rolling shutter (RS) CMOS sensors equip virtually all consumer electronics, from smartphones and tablets to action cameras and commercial drones. Unlike global shutter (GS) sensors where all pixels are exposed simultaneously, an RS sensor reads out the image row by row over an interval of 15–30 ms. When the camera moves during sensor readout, every row captures the physical environment from a slightly distinct viewpoint. This continuous camera motion violates the static pinhole camera model that underpins classical multi-view geometry, transforming the standard constant essential matrix into a dynamic, row-dependent matrix that introduces systematic biases into structure-from-motion (SfM) and visual SLAM pipelines.

Under the widely adopted first-order RS motion model, the relative pose estimation problem comprises 17 degrees of freedom: 5 for relative pose \((R, t)\) and 12 for the angular and translational velocities of both cameras \((\omega_k, v_k)\). State-of-the-art point-based solvers, such as Dai et al., require at least 20 point correspondences (PCs) for the linearized model (and up to 44 PCs for quadratic formulations). In robust estimation frameworks, RANSAC iteration count scales exponentially with the minimal sample size \(k\) as \(\eta^{-k}\). At an inlier ratio of 50%, a 5-point GS solver requires only 32 iterations, whereas a 20-point solver demands more than \(10^6\) iterations, rendering RS-aware estimation computationally prohibitive in practice. Furthermore, point constraints alone suffer from severe ill-conditioning between the translational velocity \(v\) and the baseline direction \(t\), causing translational velocity estimates to diverge wildly.

The key insight of this paper is that local affine correspondences (ACs) — which capture local \(2 \times 2\) patch deformation gradients readily available from modern feature matchers — supply 3 independent constraints per match instead of 1. Core idea: integrate affine correspondences into rolling shutter two-view geometry by deriving RS-corrected affine constraints that account for row-dependent essential matrix gradients, eliminate the 12 RS motion unknowns via null-space projection under small-motion linearization, and solve the remaining 5-DoF pose system using an action matrix algebraic solver from only 7 ACs.

Method

Overall Architecture

The pipeline consists of RS-corrected affine constraint formulation, small-motion linearization around the static point \(\theta = 0\), null-space elimination of 12 RS motion unknowns, Cayley parameterization with minor-based reduction to a degree-20 polynomial ideal, action matrix eigenvalue extraction, and non-linear joint Levenberg–Marquardt refinement within SupeRANSAC. The input is a minimal sample of 7 affine correspondences \((q_1, q_2, A_c)\) with normalized row coordinates \((\tau_1, \tau_2)\), and the output is the relative pose \((R, t)\) alongside the per-frame camera velocities \((\omega_1, v_1, \omega_2, v_2)\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 7 Affine Correspondences (q1, q2, Ac)<br/>and normalized rows (τ1, τ2)"] --> B["RS-Corrected Affine Differential Constraints<br/>Epipolar constraint + horizontal/vertical derivatives"]
    B --> C["Linearization & Null-Space Projection<br/>Eliminate 12 RS unknowns at fixed reference Rref = I"]
    C --> D["Cayley Parameterization & Minor Elimination<br/>5×3 quadratic matrix reduced by Cayley denominator"]
    D --> E["40×56 Macaulay Action Matrix Solver<br/>Extract 20 solution candidates and back-substitute RS unknowns"]
    E --> F["SupeRANSAC Scoring & Joint LM Refinement<br/>Minimize combined epipolar-affine cost with velocity damping"]
    F --> G["Output: Relative Pose (R, t)<br/>and Motion Velocities (ω, v)"]

Key Designs

1. RS-Corrected Affine Differential Constraints: Capturing Row-Dependent Matrix Gradients

In GS geometry, the essential matrix \(E\) is constant across the entire image. In contrast, under rolling shutter readout, the essential matrix \(\tilde{E}(\tau_1, \tau_2)\) is row-dependent. Differentiating the epipolar constraint \(q_2^\top \tilde{E}(\tau_1, \tau_2) q_1 = 0\) with respect to the coordinates of \(q_1\) reveals that point perturbations induce vertical shifts that alter the readout scanlines: a horizontal perturbation \(q_{1,x}\) does not change its own row \(\tau_1\) but induces a vertical displacement on \(q_2\) via the local affine transformation \(A_c\), changing \(\tau_2\); a vertical perturbation \(q_{1,y}\) directly shifts \(\tau_1\) and simultaneously shifts \(\tau_2\) via \((a_v)_y\). Applying the chain rule produces the exact RS-corrected affine constraints:

\[q_2^\top \tilde{E} e_u + a_u^\top \tilde{E} q_1 + q_2^\top \frac{\partial \tilde{E}}{\partial \tau_2} q_1 \cdot \frac{f_y}{h} (a_u)_y = 0\]

\[q_2^\top \tilde{E} e_v + a_v^\top \tilde{E} q_1 + q_2^\top \frac{\partial \tilde{E}}{\partial \tau_1} q_1 \cdot \frac{f_y}{h} + q_2^\top \frac{\partial \tilde{E}}{\partial \tau_2} q_1 \cdot \frac{f_y}{h} (a_v)_y = 0\]

where \(e_u, e_v\) are standard basis vectors, \(a_u, a_v\) are the column vectors of \(A_c\), \(f_y\) is the focal length, and \(h\) is the sensor height. Each AC supplies 3 equations (1 epipolar and 2 affine derivatives). Thus, 7 ACs yield 21 equations for the 17 unknowns. Critically, the row derivatives \(\partial \tilde{E}/\partial \tau_k\) capture spatial gradients of the motion field, providing the mathematical discriminator needed to distinguish translation \(t\) from translational velocity \(v\).

2. Physical Small-Motion Linearization and Null-Space Elimination: Decoupling RS Unknowns

Directly solving the 17-DoF non-linear system is algebraically intractable, as even the known-pose subproblem with 12 RS unknowns alone admits 254 isolated solutions. However, for real-world platforms, sensor readout is brief (\(15–30\) ms), restricting cumulative angular rotation to \(\|\omega\| \le 0.15\) rad. The quadratic terms in the epipolar formulation satisfy \(\tau^2 \|\omega\|^2 \le 5 \times 10^{-3}\), which fall well below image feature measurement noise. Linearizing the 21 constraints in \(\theta = (\omega_1, v_1, \omega_2, v_2)\) around \(\theta = 0\) yields a clean linear system:

\[J(R, t) \theta = -r_0(R, t)\]

where \(J \in \mathbb{R}^{21 \times 12}\) is the Jacobian matrix and \(r_0\) collects the GS algebraic residuals. Since \(21 > 12\), the system is overdetermined in \(\theta\). A consistent solution exists if and only if \(r_0(R, t)\) lies in the column space of \(J\), meaning \(r_0\) must be orthogonal to the left null-space basis \(U_\perp \in \mathbb{R}^{21 \times 9}\). To eliminate non-linear pose coupling in \(U_\perp\), the author computes and orthonormalizes \(U_\perp\) once at a fixed reference rotation \(R_{\text{ref}} = I\), successfully projecting out all 12 RS parameters into 9 consistency conditions \(U_\perp^\top r_0(R, t) = 0\) over pose alone.

3. Cayley Parameterization and Minor Elimination: Constructing a Degree-20 Polynomial System

Setting translation up to scale as \(t = (1, u, v)^\top\), the linearity of \(r_0\) with respect to \(t\) enables factoring the 9 equations as \(G_{\text{poly}}(s) (1, u, v)^\top = 0\), where rotation is parameterized via Cayley vectors \(s = (s_1, s_2, s_3)\) and entries of \(G_{\text{poly}}\) are degree-2 polynomials. A non-trivial translation vector exists if and only if \(G_{\text{poly}}\) has rank at most 2.

The author forms 5 random Gaussian linear combinations of the 9 rows, producing a \(5 \times 3\) polynomial matrix \(\hat{G}(s)\). All \(\binom{5}{3} = 10\) maximal \(3 \times 3\) minors must vanish. While each determinant has degree 6, the author proves that every minor is identically divisible by the Cayley denominator \(d(s) = 1 + \|s\|^2\) (since the Cayley rotation degenerates at \(d(s)=0\), forcing \(\hat{G}\) to drop rank). Factoring out \(d(s)\) yields 10 polynomial equations of degree 4 in \(s\). The Hilbert function of this polynomial ideal stabilizes at degree 20, proving that the minimal system has exactly 20 isolated algebraic solutions.

4. Action Matrix Eigenvalue Formulation and Damped Joint LM Refinement

At extension degree \(d_{\text{ext}} = 5\), a \(40 \times 56\) Macaulay matrix is constructed. SVD analysis displays a sharp spectral gap at the 36th singular value (\(\sigma_{36}/\sigma_{37} > 10^3\)), extracting the 20-dimensional quotient algebra basis \(V_{\text{null}} \in \mathbb{R}^{56 \times 20}\). Commuting action matrices \(M_k = V_{\text{null}}^\top S_k V_{\text{null}}\) are constructed for each coordinate \(s_k\) and simultaneously diagonalized to yield all 20 solution candidates in just 1.2 ms. Back-substitution recovers \(t\) and the 12 RS motion parameters via linear least squares.

During SupeRANSAC, candidate hypotheses are scored using the RS Sampson distance. The best inlier set undergoes non-linear Levenberg–Marquardt optimization minimizing the joint objective:

\[C = \sum_{i=1}^{N_{\text{in}}} \left( r_{0,i}^2 + w_{\text{ac}} r_{1,i}^2 + w_{\text{ac}} r_{2,i}^2 \right)\]

where \(r_{0,i}\) is the epipolar residual and \(r_{1,i}, r_{2,i}\) are the affine residuals. To resolve the ill-conditioned \(v-t\) ambiguity along the baseline direction, a mild Tikhonov damping term \(\lambda_v I\) is applied to the velocity blocks. Supported by the spatial derivatives from affine residuals, the optimization stably decouples camera translation from physical translational velocity.

Key Experimental Results

Main Results

Evaluation was conducted on the TUM-RS benchmark (10 fisheye sequences with motion-capture ground truth) across strides 10 and 20. Pose accuracy is quantified by the Area Under the Curve (AUC) of cumulative error curves up to 5°, 10°, and 20° thresholds on \(\max(\epsilon_R, \epsilon_t)\). RS motion accuracy is measured by the median \(L_2\) errors of angular velocity \(\epsilon_\omega\) (rad) and translational velocity \(\epsilon_v\) (m). Baselines include point-based GS (GS-5PC), affine GS (GS-2AC), and state-of-the-art point-based RS solvers (RS-20PC and RS-44PC).

Feature Extractor	Method	stride=10 AUC@5°↑	stride=10 AUC@10°↑	stride=10 \(\epsilon_\omega\) (rad)↓	stride=10 \(\epsilon_v\) (m)↓	RANSAC Time (s)↓
AffNet (Sparse)	RS-20PC [17]	0.101	0.181	0.177	11.260	12.7
AffNet (Sparse)	RS-44PC [17]	0.267	0.366	0.097	13.110	25.2
AffNet (Sparse)	GS-5PC [29]	0.428	0.535	–	–	1.5
AffNet (Sparse)	GS-2AC [7]	0.309	0.490	–	–	0.3
AffNet (Sparse)	Proposed (RS-7AC)	0.502	0.609	0.047	0.056	3.7
RoMa (Dense)	RS-20PC [17]	0.769	0.869	0.063	8.011	20.9
RoMa (Dense)	RS-44PC [17]	0.870	0.929	0.050	9.537	46.3
RoMa (Dense)	GS-5PC [29]	0.861	0.908	–	–	5.3
RoMa (Dense)	GS-2AC [7]	0.859	0.926	–	–	1.3
RoMa (Dense)	Proposed (RS-7AC)	0.897	0.948	0.043	0.051	7.5

Ablation & Generalization (EuRoC MAV Global Shutter Sequences)

To test whether the RS solver degrades when no rolling shutter distortion is present, experiments were conducted on the EuRoC MAV benchmark (6 indoor global-shutter sequences). In this regime, true RS parameters are zero.

Configuration / Method	stride=10 AUC@5°↑	stride=10 AUC@10°↑	\(\epsilon_\omega\) (rad)↓	\(\epsilon_v\) (m)↓	Note
RS-20PC (Point-based)	0.426	0.631	0.075	11.762	Large sample size degrades RANSAC inliers; velocity explodes
RS-44PC (High-order)	0.499	0.687	0.041	19.296	Suffers from baseline-velocity ambiguity; spurious large velocities
GS-5PC (Native GS)	0.521	0.704	–	–	Tailored for GS; cannot estimate velocity
GS-2AC (Affine GS)	0.517	0.702	–	–	Fast 2-AC solver assuming static essential matrix
Proposed (RS-7AC)	0.512	0.701	0.016	0.035	Matches GS solver accuracy; recovers near-zero velocity parameters

Key Findings

• Resolution of the \(v-t\) Coupling: On TUM-RS, point-based methods (RS-20PC / RS-44PC) exhibit massive velocity errors (\(\epsilon_v \approx 8–13\) m), worse than predicting zero motion. RS-7AC leverages row-gradient affine constraints and damped LM to reduce translational velocity error to \(0.051–0.056\) m — an improvement of two orders of magnitude.
• RANSAC Practicality: Slashing sample size from 20/44 to 7 boosts all-inlier probability at 30% outliers from \(\sim 10^{-7}\) to \(0.08\). Despite a 1.2 ms algebraic solver, full RANSAC runs in 3.7–7.5 s, outperforming RS-44PC by over \(6\times\) in speed while providing superior pose AUC.
• Graceful GS Degradation: On EuRoC MAV global shutter imagery, RS-7AC matches the pose AUC of standard 5-point solvers (0.512 vs. 0.521 AUC@5°) while accurately converging to near-zero RS motion (\(\epsilon_\omega \approx 0.016\) rad, \(\epsilon_v \approx 0.035\) m) without hallucinating motion artifacts.

Highlights & Insights

• Differential Calculus over Rolling Exposure: The derivation shows that perturbing a point's image coordinate along the vertical axis directly shifts its exposure timestamp, introducing an explicit \(\partial \tilde{E}/\partial \tau_k\) correction. This elegantly captures the spatio-temporal coupling in closed-form algebraic equations.
• Physics-Informed Null-Space Projection: Instead of attacking an intractable 17-unknown polynomial system, the formulation exploits the physical smallness of inter-frame motion (\(\tau^2 \|\omega\|^2 \le 5 \times 10^{-3}\)) to eliminate all 12 velocity unknowns at \(R_{\text{ref}}=I\), yielding a compact 5-row polynomial system of algebraic degree 20.
• Zero-Cost ACs from Modern Matchers: By computing numerical Jacobians of continuous warp fields via central finite differences, dense neural matchers like RoMa provide one AC per correspondence at negligible computational cost, eliminating the historical hurdle of affine feature extraction.

Limitations & Future Work

• Breakdown under Extreme Rotation: If the camera experiences angular velocities exceeding \(15^\circ–20^\circ\) during a single sensor readout, the first-order Taylor approximation incurs larger truncation residuals, which may degrade null-space projection accuracy. Incorporating high-rate gyroscope priors could mitigate this issue.
• Noise Sensitivity in Low-Texture Regions: Affine correspondences rely on accurate local gradients. In textureless regions, severe motion blur, or repetitive patterns, numerical differentiation of the correspondence field may introduce noise into \(A_c\), requiring robust affine confidence weighting.

Related Work & Insights

• vs Dai et al. (CVPR 2016 RS-20PC / RS-44PC): Dai et al. rely exclusively on point correspondences, requiring 20 to 44 points and failing to decouple translational velocity from baseline translation. The proposed method uses 7 ACs, runs drastically faster in RANSAC, and recovers precise velocities.
• vs Barath & Hajder (TIP 2018 GS-2AC): GS-2AC requires only 2 ACs but assumes a global shutter. Under rapid camera motion, it incurs large systematic errors, whereas RS-7AC models RS distortion and gracefully generalizes back to GS.
• vs Hruby & Pollefeys (ICCV 2025 Line-based RS Solver): Hruby & Pollefeys utilize single-scanline line correspondences, which are difficult to extract in uncurated outdoor environments. RS-7AC operates directly on standard feature points and dense correspondence fields.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First formulation of affine differential geometry under rolling shutter cameras and derivation of a 7-AC minimal solver.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on TUM-RS real data, synthetic parameter sweeps, and EuRoC GS zero-motion verification.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation combined with clear geometric intuition.
• Value: ⭐⭐⭐⭐⭐ Solves the longstanding computational barrier preventing RS relative pose estimation from practical real-time deployment.