PRISM-VO: Scale-Aware Visual Odometry Using Photometric Plenoptic Bundle Adjustment¶
Conference: ECCV 2026
Paper: ECCV Official
Project: https://prism-vo.github.io/
Area: 3D Vision
Keywords: Plenoptic Camera, Light Field, Visual Odometry, Photometric Bundle Adjustment, Scale Estimation
TL;DR¶
Tailored to the optical physics of focused plenoptic cameras, PRISM-VO formulates an end-to-end photometric plenoptic bundle adjustment framework that jointly optimizes single-frame micro-lens disparity depth priors and sliding-window multi-view temporal photometric consistency, enabling drift-resilient, metric-scale visual odometry without external sensors.
Background & Motivation¶
In autonomous robot navigation, self-driving vehicles, and virtual/augmented reality systems, real-time six-degree-of-freedom pose estimation and 3D mapping are essential perception capabilities. Conventional monocular cameras are widely deployed due to their lightweight, low cost, and high spatial resolution; however, the fundamental geometry of perspective projection induces inherent scale ambiguity and progressive scale drift in monocular visual odometry systems such as ORB-SLAM, DSO, and DPVO, preventing them from recovering absolute metric scale without auxiliary sensors. While stereo pairs, RGB-D sensors, and Time-of-Flight (ToF) cameras provide scale information, they remain fundamentally constrained by fixed physical baselines, limited active infrared range, vulnerability to outdoor ambient sunlight, or bulky form factors and power consumption, making them poorly suited for compact robotic payloads.
A focused plenoptic camera (plenoptic camera 2.0) introduces a micro-lens array (MLA) between the main lens and the image sensor, capturing both the spatial position and angular direction of incident light rays within a single exposure. This computational imaging setup yields thousands of micro-images, providing a very wide depth of field alongside multi-view micro-baseline observations that permit direct geometric depth estimation from a single camera body. Nonetheless, prior plenoptic visual odometry methods, such as SPO, operate strictly by performing semi-dense direct alignment across consecutive frames without multi-frame temporal bundle adjustment, leaving them vulnerable to error accumulation and tracking drift during agile motion or challenging texture conditions. Meanwhile, standard multi-view photometric bundle adjustment methods based on pinhole or fisheye models fundamentally break down under the non-single-center projection geometry of plenoptic optics.
The central tension lies in establishing a unified nonlinear optimization framework that explicitly respects the complex non-single-perspective plenoptic projection while probabilistically fusing the instantaneous metric scale of micro-lens disparities with the large temporal baseline of multi-frame photometric constraints. Core idea: by explicitly modeling the virtual image projection geometry of focused plenoptic cameras, PRISM-VO unifies Gaussian inverse virtual depth priors with multi-frame sliding-window photometric residuals into a joint nonlinear least-squares energy function, achieving drift-resilient, metric-scale visual odometry via curvature-adaptive cross-modal residual balancing.
Method¶
Overall Architecture¶
The system architecture of PRISM-VO couples a direct plenoptic front-end tracker with a back-end sliding-window plenoptic bundle adjustment. Given raw plenoptic images from a focused plenoptic camera, the preprocessing pipeline synthesizes totally focused images, virtual depth maps, and virtual depth uncertainty maps. The front-end directly aligns multi-scale image pyramids against reference keyframes in a coarse-to-fine manner to deliver initial camera poses. The back-end maintains an active sliding window of keyframes, samples salient points with high image gradients and low depth uncertainty, and jointly optimizes multi-view photometric residuals alongside plenoptic inverse virtual depth residuals through a sparse Gauss-Newton formulation with Schur complement marginalization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Plenoptic Image Input<br/>Focused Light Field Capture"] --> B["Multi-Scale Plenoptic Image & Depth Pyramids<br/>Variance-weighted downsampling & uncertainty propagation"]
B --> C["Adaptive Inverse Depth Cutoff & Point Selection<br/>Gradient diversity & inverse depth variance weighting"]
C --> D["Coarse-to-Fine Direct Photometric Tracking<br/>Totally focused image multi-scale alignment"]
D --> E["Photometric Plenoptic Bundle Adjustment<br/>Joint photometric and plenoptic depth modeling"]
E --> F["Curvature-Adaptive Cross-Modal Residual Weighting<br/>Gauss-Newton curvature matching & log-domain smoothing"]
F --> G["Scale-Aware Metric Trajectory & Sparse Map<br/>Schur complement marginalization of old keyframes"]
Key Designs¶
1. Multi-Scale Plenoptic Image and Uncertainty Pyramids: Preserving Gaussian Disparity Statistics
To extend the convergence radius of direct photometric alignment, multi-scale image pyramids are indispensable. However, unlike standard perspective images, the virtual depth \(v = B/b\) of a focused plenoptic camera (where \(B\) is the MLA-to-sensor distance and \(b\) is the MLA-to-virtual-image distance) exhibits unique statistical properties: the inverse virtual depth \(\rho_v = 1/v\) is strictly proportional to sub-pixel micro-image disparity and follows a normal distribution under sensor noise. To preserve this metric uncertainty through pyramid downsampling, totally focused images are coarsened by standard \(2 \times 2\) pixel averaging, whereas inverse virtual depth maps are downsampled via variance-weighted averaging, assigning larger influence to reliable measurements:
The coarse-level inverse virtual depth variance \(\bar{\sigma}_{\rho_{v,i}}^2\) is propagated via the harmonic mean over valid child pixels: \(\bar{\sigma}_{\rho_{v,i}}^2 = |\mathcal{N}_x^{(i)}| / \sum_{k \in \mathcal{N}_x^{(i)}} (\sigma_{\rho_{v,k}}^2)^{-1}\). This prevents high-uncertainty outliers from corrupting coarser levels and maintains statistically principled uncertainty estimates throughout the entire scale space.
2. Adaptive Depth Cutoff and Oriented Gradient Point Selection: Focusing on High-SNR Geometry
Because micro-lens baselines are extremely narrow, disparity sensitivity drops quadratically with scene distance, causing virtual depth variance to escalate rapidly for distant structures. Uniform edge sampling would inevitably introduce noisy depth priors that corrupt absolute scale estimation. PRISM-VO addresses this by deriving an adaptive percentile cutoff from the mean inverse virtual depth \(\rho_{v,\text{av}}\) across all valid pixels. The normalized cutoff weight \(w_{\text{cut}} = (\rho_{v,\text{av}} - \rho_{v,\text{f}}) / (\rho_{v,\text{n}} - \rho_{v,\text{f}})\) dynamically scales the percentile threshold \(p_a\) within preset bounds \([p_{\min}, p_{\max}]\):
During point selection, the gradient magnitude \(\nabla I(\mathbf{x}_V)\) is projected onto a randomly sampled unit direction \(\mathbf{d}\) to promote directional diversity across tracked edges. A depth weight factor \(w_d(\mathbf{x}_V)\) then heavily upweights points with verified low-variance depth while retaining unmeasured points at lower weights. This guarantees broad visual tracking coverage across the frame while anchoring metric scale firmly to high-SNR proximal structures.
3. Photometric Plenoptic Bundle Adjustment: Unifying Large Baselines and Micro-Disparities
The state vector incorporates sliding-window keyframe poses \(\boldsymbol{\xi} \in \mathfrak{se}(3)\), affine brightness parameters \(a, b\), and inverse point depths \(\rho_C\) defined in the camera coordinate frame. A virtual image point \(\mathbf{x}_{V,i}\) in host frame \(i\) is transformed into target frame \(j\) via the plenoptic projection function \(\Pi_{\text{pl}}\) and its inverse \(\Pi_{\text{pl}}^{-1}\):
Beyond the photometric residual \(r^{\text{photo}}\) (which accounts for exposure variation via affine parameters), the optimization incorporates an explicit inverse virtual depth residual \(r^{\text{depth}} = \rho_v - \rho_v^{\text{meas}}\). Because inverse virtual depth measurement noise is Gaussian, its quadratic residual is weighted directly by the inverse variance \(w_l^{\text{depth}} = (\sigma_{\rho_{v,l}}^2)^{-1}\). Crucially, \(r^{\text{depth}}\) depends exclusively on the inverse point depth \(\rho_C\) and has zero Jacobians with respect to camera poses \(\boldsymbol{\xi}\) and brightness parameters. Consequently, incorporating metric geometric priors leaves the block-arrowhead sparsity pattern of the Gauss-Newton Hessian intact, preserving the full efficiency of Schur complement elimination and marginalization.
4. Curvature-Adaptive Cross-Modal Weighting: Balancing Photometric and Geometric Constraints
Photometric residuals (intensity differences) and geometric depth residuals (inverse depth discrepancies) operate with different physical dimensions and gradient scales. In close-range scenes, micro-lens disparities are strong and geometric constraints should dominate; in distant open scenes, micro-disparities vanish into noise and temporal photometric alignment must lead. PRISM-VO introduces a dynamic scalar weight \(\eta\) derived from the ratio of average Gauss-Newton curvatures (trace of \(\mathbf{J}^\top \mathbf{J}\)) across both residual modalities:
To prevent numerical oscillations from unsettling trajectory estimates, the weight is smoothed across frames via an exponential moving average in the logarithmic domain: \(\eta_{k+1} = \exp\left( (1-\alpha) \ln \eta_k + \alpha \ln \bar{\eta} \right)\). This enables self-tuning, scene-adaptive transitions between geometric scale anchoring and temporal photometric tracking without manual hyperparameter tuning.
Loss & Training¶
PRISM-VO is a purely optimization-based estimation pipeline and requires no neural network training. The total energy minimized across the active sliding window combines robustly weighted photometric and inverse virtual depth residuals:
Here, \(\|\cdot\|_\gamma\) denotes the Huber norm to suppress gross outliers caused by dynamic occlusions or non-Lambertian reflections; \(w_k^{\text{photo}}\) incorporates local image gradient weights; and \(w_l^{\text{depth}} = (\sigma_{\rho_{v,l}}^2)^{-1}\) enforces variance-proportional weighting. State increments are solved iteratively via damped Levenberg-Marquardt steps \((\mathbf{H} + \mu \mathbf{I}) \delta \mathbf{s} = -\mathbf{b}\). Outgoing keyframes and inactive landmark observations are marginalized via the Schur complement to carry historical geometric information forward as Gaussian priors.
Key Experimental Results¶
Main Results¶
PRISM-VO was extensively evaluated on two benchmark datasets: the Synchronized Stereo and Plenoptic Visual Odometry Dataset (Raytrix R5 focused plenoptic camera, comprising 11 loop-closure sequences spanning hundreds of meters indoors and outdoors) and the LiFMCR Dataset (two high-resolution Raytrix R32 cameras with sub-millimeter 6-DoF Vicon ground truth).
On the 11 long-loop sequences of the synchronized dataset, cumulative drift was quantified via start-and-end \(\mathrm{Sim}(3)\) alignment to extract absolute scale error \(d'_s = \max\{d_s, d_s^{-1}\}\), scale drift \(e'_s = \max\{e_s, e_s^{-1}\}\), overall trajectory alignment error \(e_{\text{align}}\), and rotation error \(e_r\). The table below reports the number of sequences reaching high / medium / coarse precision thresholds against the state-of-the-art plenoptic baseline SPO:
| Metric & Precision Threshold | PRISM-VO (Ours) | SPO (Prior SOTA Plenoptic VO) | Note |
|---|---|---|---|
| Absolute scale error \(d'_s \le (1.05 / 1.10 / 1.30)\) | 5 / 8 / 10 | 5 / 7 / 9 | Tight scale accuracy (\(d'_s \le 1.10\)) achieved on 8/11 sequences |
| Scale drift \(e'_s \le (1.05 / 1.10 / 1.30)\) | 7 / 7 / 10 | 7 / 8 / 9 | Matches SPO at high precision, superior overall sequence coverage |
| Trajectory alignment error \(e_{\text{align}} \le (1\% / 2\% / 4\%)\) | 4 / 6 / 10 | 2 / 6 / 9 | High-precision sequence count doubled (4 vs 2), reducing drift |
| Accumulated rotation error \(e_r \le (1^\circ / 2^\circ / 4^\circ)\) | 6 / 8 / 10 | 3 / 7 / 8 | Substantial rotation consistency gain (\(e_r \le 1^\circ\) on 6 vs 3 runs) |
On the LiFMCR dataset across 7 close-range object-inspection trajectories, PRISM-VO was benchmarked against DSO, monocular ORB-SLAM3, and learning-based DPVO (translational RMSE \(\text{RMSE}_t\) in mm, rotational RMSE \(\text{RMSE}_r\) in \(^\circ\)):
| Scene ID & Name | PRISM-VO (Ours) \(\text{RMSE}_t\) / \(\text{RMSE}_r\) | DSO \(\text{RMSE}_t\) / \(\text{RMSE}_r\) | ORB-SLAM3 (mono) \(\text{RMSE}_t\) / \(\text{RMSE}_r\) | DPVO \(\text{RMSE}_t\) / \(\text{RMSE}_r\) |
|---|---|---|---|---|
| 01 Plants | 41.66 / 2.98 | 59.32 / 48.56 | 271.24 / 98.79 | 607.44 / 3.62 |
| 02 Bike | 57.81 / 6.09 | 112.56 / 21.24 | 316.78 / 25.66 | 356.60 / 2.60 |
| 03 Lab | 38.34 / 2.86 | 326.70 / 1.97 | 123.20 / 10.71 | 169.23 / 2.04 |
| 04 Electronics | 10.53 / 2.83 | 150.89 / 87.20 | 261.02 / 45.97 | 268.47 / 3.01 |
| 05 Desk | 9.32 / 2.89 | 239.16 / 9.15 | 121.64 / 56.93 | 219.66 / 2.92 |
| 06 Tools | 12.21 / 3.31 | 543.90 / 164.22 | 773.65 / 153.28 | 704.67 / 163.09 |
| 07 Machinery | 110.87 / 11.86 | 253.36 / 162.17 | 225.55 / 21.12 | 5.87 / 1.99 |
Ablation Study¶
An incremental ablation on the synchronized plenoptic dataset demonstrates the specific performance contributions of each design component:
| Config | Core Modification | Scale Observability | Drift & Trajectory Consistency |
|---|---|---|---|
| (1) Pinhole Baseline | Central perspective model ignoring plenoptic optics and micro-disparities | None (arbitrary scale) | Severe scale drift, slow alignment convergence |
| (2) Plenoptic Model & Initialization | Plenoptic projection \(\Pi_{\text{pl}}\) and depth initialization, without depth residuals | Absolute metric scale recovered | Noticeable scale drift reduction; tracking errors accumulate |
| (3) Full PRISM-VO System | Joint inverse virtual depth residuals with variance weighting and adaptive \(\eta\) | Fully metric and robust | Optimal alignment and scale consistency across all sequences |
Key Findings¶
- Synergy of micro-disparities and temporal baselines: The instantaneous baseline between adjacent micro-lenses provides reliable absolute metric scale within proximal ranges, while temporal camera motion provides meter-scale baselines that strongly constrain orientation and distant structure. Their unified optimization suppresses the drift vulnerabilities of both isolated paradigms.
- Critical role of variance weighting and adaptive curvature balancing: Downsampling depth maps using inverse-variance weights and balancing cross-modal residuals via Gauss-Newton curvature matching prevents noisy distant depth estimates from degrading tracking while preventing dense close-range depth residuals from overwhelming photometric tracking.
- Robustness in challenging optical conditions: In sequences featuring severe motion blur, fine intersecting fan grilles, semi-transparent blades, and reflective metallic surfaces, conventional methods (ORB-SLAM3, DSO) fail due to tracking loss, and active RGB-D sensors (RealSense D455) exhibit major depth voids. PRISM-VO successfully maintains continuous tracking with millimeter-level translational precision.
Highlights & Insights¶
- Photometric bundle adjustment beyond central perspective models: While classical direct SLAM formulations are strictly bound to pinhole or fisheye assumptions, PRISM-VO demonstrates the exact analytic integration of non-single-perspective plenoptic projection into sliding-window bundle adjustment.
- Zero-fill-in Hessian preservation for depth priors: Formulating the geometric prior in inverse virtual depth space decouples landmark depth errors from camera motion and illumination parameters, keeping the Gauss-Newton Hessian block-arrowhead structure intact and allowing fast Schur complement marginalization.
- Curvature-guided multi-modal residual weighting: Using the ratio of Gauss-Newton curvatures combined with log-domain smoothing provides an elegant, self-tuning mechanism to balance heterogeneous residual types without empirical hyperparameter sweeps.
Limitations & Future Work¶
- Reduced spatial resolution and narrow field of view: Due to the trade-off of dedicating sensor pixels to angular micro-images, plenoptic cameras offer lower spatial resolution and a narrower field of view compared to standard cameras of equivalent sensor size.
- Depth range limitation: As distance increases, micro-lens disparity shrinks below sub-pixel noise thresholds, causing the system to transition smoothly into a weakly-scale-observable state dominated by temporal photometric constraints.
- Future directions: Integrating learned light-field feature correspondences for wide-baseline relocalization, and tightly coupling plenoptic bundle adjustment with an inertial measurement unit (VIO) to maintain scale observability in long-range open spaces.
Related Work & Insights¶
- vs SPO (Semi-dense Plenoptic Odometry): SPO aligns consecutive micro-images without multi-frame temporal bundle adjustment; PRISM-VO introduces a keyframe-based sliding-window optimization with variance-weighted depth residuals, cutting rotational drift and alignment errors by more than half while succeeding on sequences where SPO fails.
- vs DSO (Direct Sparse Odometry): DSO operates on monocular pinhole optics and produces reconstructions up to an unknown global scale factor; PRISM-VO embeds the plenoptic camera model and inverse virtual depth priors into the photometric optimization, recovering absolute metric scale natively.
- vs DPVO / DROID-SLAM: Learning-based VO architectures leverage extensive prior training but cannot directly handle raw micro-lens arrays without retraining; PRISM-VO relies entirely on physics-based optimization, offering full interpretability and native metric scale recovery from raw plenoptic optics.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first sparse photometric bundle adjustment pipeline tailored to focused plenoptic cameras with principled uncertainty propagation and adaptive residual balancing.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated across both hundred-meter outdoor loop sequences and millimeter-accurate indoor motion-capture benchmarks with extensive sensor comparisons and ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical formulations, rigorous geometric derivations, and well-structured comparative narratives.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational framework for scale-aware, single-sensor metric visual odometry in compact robotics and medical endoscopy.