Rolling Shutter Camera Self-Calibration¶

Conference: ECCV 2026
Paper: ECCV Full Paper
Code: https://github.com/Yongcong-Zhang/RSSC
Area: 3D Vision
Keywords: Rolling Shutter, Camera Self-Calibration, Bundle Adjustment, B-Spline Continuous Trajectory, Dual-Projection Model

TL;DR¶

The first target-free and IMU-free visual self-calibration framework for rolling shutter cameras, unifying continuous-time B-spline trajectory estimation and 2D pixel trajectory rectification into a dual-projection bundle adjustment to jointly estimate camera intrinsics and readout time ratio from image sequences.

Background & Motivation¶

Rolling shutter (RS) cameras are ubiquitous in modern consumer devices, including smartphones, drones, autonomous vehicles, and mobile robotic platforms, owing to their cost-effectiveness, low power consumption, and high frame rates. Unlike global shutter (GS) sensors where all pixels are exposed simultaneously at a single instant, an RS sensor exposes and reads out image rows sequentially in a row-by-row manner. As a result, each image row captures the 3D scene at a slightly different timestamp. Under camera ego-motion or dynamic scene movement, this continuous temporal delay introduces characteristic geometric distortions across frames, such as skew, wobble, and the infamous jello effect. These non-rigid distortions fundamentally violate the classical rigid-body epipolar geometry and perspective projection assumptions underpinning Structure from Motion (SfM), visual SLAM, and Bundle Adjustment (BA).

Accurate geometric 3D vision strictly requires precise camera calibration. For RS cameras, the imaging projection depends not only on classical intrinsic parameters (focal length and principal point) but also crucially on the readout time ratio $\gamma$—defined as the ratio of the active row-wise exposure readout duration to the inter-frame temporal interval. The readout time ratio $\gamma$ directly dictates the amplitude of non-rigid image distortions, making it essential for RS image rectification and motion estimation. However, existing RS calibration approaches heavily rely on dedicated physical calibration patterns (e.g., high-frequency flashing LED panels or planar checkerboards) or tightly synchronized auxiliary sensors like Inertial Measurement Units (IMUs). In unconstrained natural environments and in-the-wild video captures, such specialized targets or hardware are rarely available, while conventional GS self-calibration methods fail catastrophically due to their inability to model row delays.

The core challenge lies in jointly decoupling camera intrinsics, continuous 6-DoF camera motion trajectories, and the temporal readout time ratio directly from image feature correspondences without any external references. The angle of attack in this work is to bridge and synthesize two historically disjoint RS modeling paradigms: continuous-time pose modeling and observation-space image rectification. Core idea: unify continuous-time cumulative B-spline trajectory estimation with 2D feature trajectory rectification into a dual-projection bundle adjustment framework, where each 3D point is simultaneously constrained by both row-dependent RS observations and reference-time rectified GS observations along a shared continuous trajectory.

Method¶

Overall Architecture¶

The proposed Rolling Shutter Camera Self-Calibration (RSSC) framework takes an unconstrained sequence of RS video frames as input. It first runs a standard GS SfM pipeline (e.g., COLMAP) to initialize a coarse reconstruction consisting of camera intrinsics $K$, discrete camera poses, and sparse 3D point coordinates $P$, with the readout time ratio $\gamma$ initialized to a default value of 0.5. Building upon this initialization, the framework introduces and harmonizes two complementary perspectives into a unified bundle adjustment optimization:

Continuous-time Trajectory Estimation (RSSC-TE): Models the RS frame as a stack of $H$ independent single-row GS images, parameterizing the continuous camera motion on Lie groups using cumulative cubic B-splines to evaluate exact row-dependent poses;
Correction Field Estimation (RSSC-CE): Interprets each RS image as a single GS image warped by an unknown 2D temporal distortion field, rectifying multi-view feature observations to a unified frame reference timestamp via temporal trajectory interpolation;
Dual-Projection Model (RSSC-DP): Eliminates independent GS pose parameters by enforcing that the reference GS poses are strictly governed by the same continuous B-spline trajectory, establishing a dual-projection reprojection error for each 3D point across both original RS and rectified GS domains.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RS Image Sequence"] --> B["Global Shutter SfM Initialization<br/>Coarse Poses / Intrinsics K / γ=0.5"]
    B --> C["Continuous-time Trajectory Estimation<br/>Lie Group Cumulative B-Spline"]
    B --> D["2D Observation Trajectory Rectification<br/>Polynomial / Hermite Temporal Interpolation"]
    C --> E["Dual-Projection Joint Optimization<br/>Shared Continuous Trajectory Dual Geometric Closed-Loop"]
    D --> E
    E --> F["Self-Calibrated Output<br/>Accurate Intrinsics K, Readout Ratio γ, and Trajectory"]

Key Designs¶

1. Continuous-time Trajectory Estimation: Decoupling Dynamic Row Poses via Physical Timestamps To handle the temporal disparity across rows, discrete frame-level rigid poses are replaced by a continuous Lie group trajectory. Denoting the row readout time as $T_{\text{readout}}$, the inter-frame non-exposure interval as $T_{\text{delay}}$, and the frame-to-frame period as $T_f = T_{\text{readout}} + T_{\text{delay}}$, the readout time ratio is defined as $\gamma = T_{\text{readout}} / T_f$. For an image of height $H$, the physical timestamp for a 2D measurement $m_i^j = [u_i^j, v_i^j]^\top$ observed at row $v_i^j$ in frame $i$ is formulated as: $$t(\gamma, v_i^j) = \left(i + \gamma \frac{v_i^j}{H}\right) T_f$$ The camera trajectory is parameterized by a set of control poses $\{T_i \in SE(3)\}$ spaced uniformly by $\Delta t = T_f$. For any time $t \in [t_i, t_{i+1})$, the continuous pose $T(t)$ is smoothly interpolated by four neighboring control poses using cumulative Lie group basis functions $B_k(s)$ and Lie algebra velocity elements $\Omega_{i-1+k} = \log(T_{i-1+k-1}^{-1} T_{i-1+k}) \in \mathfrak{se}(3)$. The corresponding row-dependent RS reprojection residual is: $$e_i^{j, \text{TE}} = m_i^j - \pi\left(T(t(\gamma, v_i^j)) P^j\right)$$ Minimizing this error across all matched points propagates sensitive analytical gradients directly to the readout time ratio $\gamma$, calibrating the temporal readout parameter along the continuous trajectory.

2. 2D Observation Trajectory Rectification: Aligning RS Points to Reference GS Timestamps When continuous 3D trajectory estimation faces non-smooth motions or jerky high-frequency vibrations, pure pose interpolation can become vulnerable to local minima. RSSC-CE tackles this from the 2D observation space by modeling the RS image as a GS image distorted by a 2D temporal motion field. By tracking the multi-frame 2D observations of the same 3D point $P^j$, its 2D coordinates are temporally interpolated to the reference timestamp $t_i = i T_f$ of frame $i$. Two interpolation schemes are formulated: - Quadratic Polynomial Interpolation (RSSC-CEQ): Employs three consecutive observations across frames $i-1, i, i+1$ using Lagrange basis functions to interpolate the rectified 2D position $m_i^{j, \text{GS}}(t_i)$; - Cubic Hermite Interpolation (RSSC-CEH): Employs four consecutive observations across frames $i-1, i, i+1, i+2$, computing numerical tangent vectors via central differences to ensure $C^1$ velocity smoothness. The rectified GS observations bypass row-wise time offsets, formulating the standard GS reprojection residual: $$e_i^{j, \text{CE}} = m_i^{j, \text{GS}}(t_i) - \pi\left({}^{\text{GS}}T_i P^j\right)$$ This observation-domain rectification operates directly in pixel space, offering rapid convergence and high initial stability.

3. Dual-Projection Joint Optimization: Unifying Spatiotemporal Motion and Observation Constraints While RSSC-TE provides strong global trajectory regularization, it struggles when actual camera motion violates the B-spline smoothness assumption. Conversely, RSSC-CE requires no global motion prior but is vulnerable to 2D measurement noise and exhibits weaker sensitivity to $\gamma$ under subtle parallax. RSSC-DP unifies both paradigms by eliminating the free GS pose parameter ${}^{\text{GS}}T_i$, enforcing that it must be strictly generated by evaluating the shared continuous B-spline trajectory at timestamp $t_i$, i.e., ${}^{\text{GS}}T_i = T(t_i)$. Under this formulation, every 3D point $P^j$ observed in frame $i$ induces two physically coupled projection events along the same continuous trajectory: a row-dependent RS projection at timestamp $t(\gamma, v_i^j)$ and a rectified GS projection at reference timestamp $t_i$. The joint objective is formulated as a weighted non-linear least-squares cost: $$\min_{\theta_{\text{DP}}} \sum_{i \in \mathcal{F}} \sum_{j \in \mathcal{P}_i} \left( \|e_i^{j, \text{TE}}\|^2 + \lambda \|e_i^{j, \text{CE}}\|^2 \right)$$ where $\theta_{\text{DP}} = \{\gamma, K, \{T_i\}, \{P^j\}\}$ and $\lambda$ balances the contributions of the two projection models (empirically set to 0.5). If 3D camera motion undergoes localized non-smooth perturbations, the 2D pixel trajectory retains local smoothness, allowing the RSSC-CE residuals to compensate for B-spline trajectory approximation errors. Conversely, when image observations are corrupted by high-frequency pixel noise, the continuous-time Lie group prior in RSSC-TE regularizes the optimization, producing a mutually reinforcing, robust spatiotemporal bundle adjustment.

Loss & Training¶

The framework is optimized end-to-end using non-linear least-squares solvers (such as damped Gauss-Newton or Levenberg-Marquardt). Parameter initialization is provided by COLMAP for $K$, $\{T_i\}$, and $\{P^j\}$, while $\gamma$ is initialized to 0.5. Optimization terminates when parameter updates drop below $10^{-8}$, the infinity norm of the gradient falls below $10^{-10}$, or the relative cost reduction is below $10^{-8}$. Due to the favorable loss landscape shaped by the dual-projection constraints, the solver converges reliably within 9 to 14 iterations.

Key Experimental Results¶

Main Results¶

Self-calibration performance was evaluated on two public real-world rolling shutter benchmarks, WHU-RSVI and TUM-RSVI, using 50 consecutive frames per sequence in unconstrained scenes without calibration boards. Since no prior visual self-calibration methods exist for RS cameras, baselines include leading global shutter calibration approaches: industrial-standard COLMAP, deep geometry-aware DroidCalib, and self-supervised SelfSup-Calib. Metrics include the Median Absolute Error (MAE) for camera intrinsics and readout time ratio $\gamma$, along with Median Absolute Trajectory Error (MATE) after Sim(3) alignment.

Dataset	Method	MAE($f_x$) (pix)	MAE($f_y$) (pix)	MAE($c_x$) (pix)	MAE($c_y$) (pix)	MAE($\gamma$)	MATE (m)
WHU-RSVI	COLMAP	3.05	11.26	4.36	8.17	—	0.0453
	DroidCalib	0.56	22.33	3.13	13.80	—	0.0317
	SelfSup-Calib	195.98	129.28	13.85	23.78	—	0.4585
	RSSC-TE	0.07	0.31	0.23	0.60	0.0485	0.0170
	RSSC-CEQ	0.16	2.82	0.27	1.04	0.0814	0.0131
	RSSC-CEH	0.67	2.43	0.71	1.60	0.1789	0.0238
	RSSC-DPQ (Ours)	0.03	0.44	0.19	0.65	0.0483	0.0101
	RSSC-DPH	0.30	0.31	0.24	1.50	0.0945	0.0126
TUM-RSVI	COLMAP	33.58	14.82	16.01	28.50	—	0.0374
	DroidCalib	71.96	67.56	17.54	41.76	—	0.0963
	SelfSup-Calib	359.77	338.17	59.89	9.85	—	0.1432
	RSSC-TE	3.16	1.48	1.76	1.12	0.1144	0.0063
	RSSC-CEQ	3.79	2.67	1.71	1.25	0.0609	0.0066
	RSSC-CEH	3.71	3.31	1.40	1.52	0.0852	0.0081
	RSSC-DPQ (Ours)	3.10	2.39	1.93	1.16	0.0814	0.0055
	RSSC-DPH	4.17	3.06	2.16	2.14	0.1374	0.0072

Ablation Study¶

The variants were thoroughly analyzed in a controlled synthetic environment across varying motion trajectory smoothness (cutoff frequency $f_c \in \{0, 2, 4, 6\}$ Hz), readout time ratios ($\gamma \in \{0.2, 0.4, 0.6, 0.8\}$), and 2D measurement noise levels ($\sigma \in \{0.5, 1.0, 1.5, 2.0\}$ pixels). Computational optimization efficiency over 50 independent trials is summarized below:

Method Variant	Core Modeling Scheme	Time per Iteration (s)	Iterations to Converge	Total Time (s)	Precision & Robustness Characteristics
RSSC-TE	Continuous B-spline trajectory only	0.5142	32	5.8657	Highly accurate under smooth motion; slow convergence; sensitive to high-frequency motion
RSSC-CEQ	Quadratic polynomial rectification only	0.0841	5	0.2321	Fastest optimization; highly vulnerable to 2D measurement noise
RSSC-CEH	Cubic Hermite rectification only	0.0851	6	0.2765	Highly efficient; higher-order derivatives introduce variance under high noise
RSSC-DPQ	Unified trajectory + Quadratic rectification	0.7668	9	2.6048	Best trade-off: Lowest errors across all noise/smoothness levels, robust intrinsic & $\gamma$ estimation
RSSC-DPH	Unified trajectory + Hermite rectification	0.7735	14	3.9185	Robust and accurate; slightly higher iteration count than polynomial version

Key Findings¶

Elimination of Geometric Drift: On WHU-RSVI, standard COLMAP exhibits large systematic errors ($f_y$ error of 11.26 pixels and $c_y$ error of 8.17 pixels) caused by absorbing temporal distortions into focal length and principal point. RSSC-DPQ reduces these errors to sub-pixel accuracy (0.03 to 0.65 pixels), achieving an order-of-magnitude precision improvement;
Accurate Target-Free Readout Estimation: Across real sequences, the proposed framework accurately estimates the unknown parameter $\gamma$ without any calibration patterns, reaching an MAE of 0.0483 on WHU-RSVI;
Complementary Regularization: Synthetic simulations demonstrate that while pure RSSC-CE error spikes exponentially under increasing 2D pixel noise (up to 2.0 pixels), RSSC-DP maintains steady, low error matching RSSC-TE due to the continuous trajectory prior. Simultaneously, under non-smooth trajectory perturbations, RSSC-DP outperforms RSSC-TE;
Optimization Acceleration: RSSC-DPQ converges in only 9 iterations (2.60 seconds total), speeding up optimization by more than $2.2\times$ compared to RSSC-TE (32 iterations, 5.87 seconds) due to effective gradient guidance from the 2D rectified observations.

Highlights & Insights¶

Pioneering Target-Free Self-Calibration: Eliminates the prerequisite of dedicated calibration hardware or flashing LED panels for RS cameras, unlocking accurate rolling shutter calibration from arbitrary natural videos;
Spatiotemporally Coupled Dual-Projection: Synthesizes continuous-time Lie group motion modeling and 2D observation-space rectification onto a single shared B-spline trajectory, creating a mutually regularizing geometric loop;
Practical Plug-and-Play Utility: Requiring only standard SfM outputs as initialization and converging in approximately 2.6 seconds, the formulation can be seamlessly incorporated into existing SLAM, SfM, and 3D reconstruction pipelines.

Limitations & Future Work¶

Degenerate Critical Motions: Specific camera trajectories—such as zero motion, constant pure rotation, or pure translation parallel to the image sensor readout lines—render the rolling shutter effect degenerate or unobservable, creating ambiguity in estimating $\gamma$. Future research should provide theoretical characterization of these critical motion sequences;
Reliance on Sparse Front-End Matching: Feature matching failures in low-texture environments or under large dynamic occlusions can introduce outliers into the 2D trajectory fitting, degrading the rectification field;
Dense Scene Representation Synergy: The authors note that an exciting future avenue is integrating this self-calibration formulation with dense 3D representations, such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS), for high-fidelity novel view synthesis from uncalibrated consumer RS videos.

vs. Target-Based RS Calibration (ARSC, RSCC): Traditional methods require high-frequency LED panels or calibrated checkerboard sequences under controlled lighting; this work enables target-free calibration in unconstrained environments;
vs. IMU-Assisted RS Calibration (Huai et al., Lee et al.): Prior target-free approaches rely on synchronized IMU data and spatial-temporal extrinsic pre-calibration; this method is purely monocular visual and hardware-free;
vs. Global Shutter Self-Calibration (COLMAP, DroidCalib, SelfSup-Calib): Standard GS self-calibration ignores row-wise exposure delays, conflating motion-induced distortions with optical lens parameters and leading to severe scale collapse; this work explicitly models physical row timestamps to decouple them.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [First purely visual, target-free, and IMU-free rolling shutter self-calibration method; elegant dual-projection formulation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive synthetic perturbations across noise, trajectory smoothness, and readout ratio, paired with comprehensive real-world benchmarks]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, crisp conceptual diagrams, and structured logical arguments]
Value: ⭐⭐⭐⭐⭐ [Removes a major barrier for deploying high-precision 3D vision, SfM, and SLAM on consumer rolling shutter cameras]