Skip to content

OmniDS: Dual-Stream Context Fusion for Omnidirectional Depth from Fisheye Cameras

Conference: ECCV 2026
Paper: ECCV 2026 Official Portal
Area: 3D Vision
Keywords: Omnidirectional depth estimation, multi-view fisheye stereo, dynamic context fusion, multi-view consensus volume, knowledge distillation

TL;DR

Addressing visibility conflicts and rigid projection ambiguities in wide-baseline multi-fisheye setups, OmniDS combines a dual-stream CNN and frozen DINOv3 dynamic ERP context fusion with a multi-view consensus volume, and achieves real-time ~10 FPS inference via lightweight MobileNet distillation.

Background & Motivation

Omnidirectional depth perception from surround-view multi-fisheye camera rigs provides 360° blind-spot-free spatial sensing in a compact form factor, serving as a core foundation for autonomous obstacle avoidance, local motion planning, and global SLAM across service robots, warehouse inspection drones, and AR/VR spatial computing. Traditional learning-based methods typically warp distorted multi-camera features onto spherical concentric sweeps or unified Equirectangular Projections (ERP) to construct geometric cost volumes, followed by 3D convolutional cost regularization or recurrent update operators (such as OmniMVS and RomniStereo).

However, wide-baseline fisheye camera rigs inherently induce severe parallax and self-occlusions. When multiple cameras observe a nearby object from distinct vantage points, they frequently capture entirely different surface faces or suffer severe sightline occlusions. Under such circumstances, rigid projection and fixed grid aggregation create fundamental matching discrepancies: even with perfectly accurate projection geometry, standard fisheye-to-ERP sampling triggers "occlusion leakage" (where an occluded camera injects background pixels into the foreground ERP cell) and "self-occlusion surface mixing" (where distinct physical faces with different surface normals and textures are forced into the same cell). Existing methods like RomniStereo down-weight unreliable views or use center-view heuristics, but this merely suppresses outliers numerically rather than untangling the underlying multi-modal feature ambiguity; moreover, pure CNN-based context models struggle with long-range structural coherence in textureless walls and reflective surfaces.

To overcome this limitation, OmniDS moves away from static ERP aggregation and instead pairs dynamic, depth-conditioned context formation with a cross-camera consensus mechanism. Core idea: construct a dual-stream encoder combining lightweight CNN features for high-frequency geometry with frozen DINOv3 features for semantic priors, dynamically reprojecting them into ERP space via learned view-weighting and distortion-biased deformable attention, coupled with a 3D-regularized multi-view consensus volume and mobile distillation for real-time edge execution.

Method

Overall Architecture

Given four calibrated 220° fisheye images \(\{I_k\}_{k=1}^4\) covering front, right, back, and left views with known optical intrinsics, OmniDS estimates an ERP inverse-depth map \(\hat{d} \in \mathbb{R}^{H \times W}\).

The pipeline executes a coordinated sequence of "dual-stream extraction → dynamic context fusion & static consensus volume construction → ConvGRU iterative refinement". First, each fisheye image passes through a shared 18-layer residual CNN to extract fine-grained geometric features \(F_k^{\mathrm{cnn}}\), while in parallel a frozen DINOv3 ViT backbone with a trainable projection head produces rich semantic features \(F_k^{\mathrm{dino}}\). Next, at each iterative refinement step \(t\), the system reprojects both features into ERP space according to the current depth estimate \(\hat{d}^{(t)}\): CNN features are aggregated with learned view weights, while DINO features are sampled through deformable cross-attention with geometric distortion bias. Concurrently, static similarity volumes—including a 2D-aggregated pairwise correlation profile and a 3D U-Net regularized multi-view consensus volume (MVC)—are indexed across discrete depth bins. A ConvGRU update block takes the concatenated dynamic context, motion features, and multi-scale volume lookups to predict residual depth updates \(\Delta d^{(t)}\). To ensure real-time mobile deployment, the dual-stream feature extractor is distilled into a single compact MobileNetV2 backbone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Four Surround Fisheye Inputs<br/>Front / Right / Back / Left"] --> B["Dual-Stream Feature Extraction & Distortion Bias<br/>CNN local geometry + frozen DINOv3 semantic priors"]
    B --> C["Multi-View Consensus Volume & 3D Regularization<br/>Group-wise correlation + 4-view feature variance + 3D U-Net"]
    B --> D["Dynamic ERP Context Fusion<br/>Iterative depth-guided reprojection + view weighting & deformable attention"]
    C --> E["ConvGRU Iterative Refinement<br/>Motion feature aggregation & residual inverse depth updates"]
    D --> E
    E --> F["Convex Upsampling Output<br/>192 discrete inverse-depth levels"]
    B -.->|Knowledge Distillation| G["Distilled Lightweight Student Encoder<br/>Unified MobileNetV2 for real-time edge deployment"]

Key Designs

1. Dual-Stream Feature Extraction & Distortion Bias: Complementing Geometric Detail with Foundation Priors Standard omnidirectional depth networks rely exclusively on convolutional features, which inevitably degrade in textureless expanses, shadows, or reflective glare. Conversely, naively deploying vision transformers tends to over-smooth high-frequency disparity transitions. OmniDS resolves this dilemma through a decoupled dual-stream design: a shared 18-layer residual CNN extracts fine-grained edge and texture representations \(F_k^{\mathrm{cnn}}\) dedicated to cost volume matching and geometric boundaries; simultaneously, a frozen DINOv3 ViT backbone outputs semantic descriptors \(F_k^{\mathrm{dino}}\) providing long-range scene understanding. To accommodate the non-uniform optical compression of fisheye lenses, where peripheral regions exhibit extreme distortion, OmniDS introduces a geometric distortion bias into the deformable cross-attention module. For each ERP query \(q_i\), \(P\) sampling offsets \(\Delta p_{i,k,j}\) around reference point \(p_{i,k}\) are modulated by an explicit radial distance bias \(b(r_{i,k}) = \mathrm{MLP}(\min(\|2 p_{i,k} - c_k\|, \dots))\). This design steers the attention mechanism to prioritize the high-density, low-distortion central field of view of each lens, mitigating semantic corruption from peripheral distortion.

2. Dynamic ERP Context Fusion: Resolving Occlusion Leakage and Surface Inconsistency In conventional pipelines, multi-view features are projected into ERP coordinates only once via static interpolation, baking occlusion artifacts permanently into the recurrent memory. In contrast, OmniDS dynamically reconstructs the ERP context at every iteration \(t\) guided by the intermediate inverse-depth prediction \(\hat{d}^{(t)}\). For CNN features, multi-camera descriptors \(F_{i,k}^{\mathrm{cnn}}\) are concatenated and passed through a lightweight view-weighting network to compute normalized coefficients \(w_{i,k} = \mathrm{Softmax}_k(\mathrm{Cam}_{\mathrm{weight}}([F_{i,1}^{\mathrm{cnn}}; \dots; F_{i,4}^{\mathrm{cnn}}]))\), yielding the aggregated feature \(F_i^{\mathrm{cnn\text{-}erp}} = \sum_{k=1}^4 w_{i,k} F_{i,k}^{\mathrm{cnn}}\). This is fused with the deformable attention-projected DINO context \(F^{\mathrm{dino\text{-}erp}}\) via residual pointwise convolution: $\(F^{\mathrm{context}} = F^{\mathrm{dino\text{-}erp}} + \mathrm{Fuse}([F^{\mathrm{dino\text{-}erp}}; F^{\mathrm{cnn\text{-}erp}}])\)$ By continuously adjusting view weights and sampling locations according to updated depth bounds, the context network dynamically filters out occluded sightlines and resolves ambiguous surface multi-modality.

3. Multi-View Consensus Volume & 3D Regularization: Multi-View Consistency and Variance Constraints Beyond pairwise correlation, multi-fisheye surround geometry requires evaluating global multi-camera consensus. OmniDS constructs a dual-signal Multi-View Consensus Volume (MVC). First, feature channels are split into 8 groups to compute group-wise correlations across views, capturing multi-modal cost distributions that exceed simple scalar dot products. Second, the framework computes the cross-camera feature variance across all 4 views at each 3D ERP-depth voxel. When a depth hypothesis matches the true surface, all visible cameras exhibit minimal feature variance, forming a sharp valley; conversely, at incorrect depths or occluded boundaries, cross-camera discrepancy causes variance to surge. The concatenated volume is regularized by a 3D U-Net operating jointly across spatial dimensions \((H, W)\) and depth planes \((D)\), suppressing local minima and false matching noise. Both the consensus volume and the 2D-smoothed correlation profiles are arranged into a 4-level feature pyramid for ConvGRU lookup.

4. Distilled Lightweight Student Encoder: Bridging Accuracy and Edge Real-Time Constraints Although the CNN + DINOv3 dual-stream architecture delivers high-accuracy representations during offline exploration, concurrently running a heavy ViT and CNN during inference exceeds the latency and memory envelopes of mobile robotics. OmniDS employs feature-level knowledge distillation: a single U-Net structured MobileNetV2 student model is trained with two dedicated \(1 \times 1\) convolution heads, generating a \(\times 2\) downsampled geometric map \(F_{\mathrm{cnn}}^s\) and a \(\times 16\) semantic map \(F_{\mathrm{dino}}^s\) (both normalized to 32 channels). The distillation objective utilizes an asymmetric mean squared error: $\(\mathcal{L}_{\mathrm{distill}} = \lambda_{\mathrm{cnn}} \mathrm{MSE}(\mathbf{F}_{\mathrm{cnn}}^s, \mathbf{F}_{\mathrm{cnn}}^t) + \lambda_{\mathrm{dino}} \mathrm{MSE}(\mathbf{F}_{\mathrm{dino}}^s, \mathbf{F}_{\mathrm{dino}}^t)\)$ with weights set to \(\lambda_{\mathrm{cnn}} = 1.0\) and \(\lambda_{\mathrm{dino}} = 10.0\) to emphasize matching the high-dimensional DINO manifold. Distillation converges in just 1 epoch on OmniThings, after which the student encoder completely supplants the dual teachers for end-to-end fine-tuning. This cuts inference latency from 148 ms to 102 ms (a 31.1% speedup) on an RTX 4070 SUPER GPU, enabling real-time operation at ~10 FPS with negligible accuracy loss.

Loss & Training

The network iteratively updates depth through \(T\) refinement steps (predicting 96 internal bins and convexly upsampling by \(2\times\) to \(N=192\) discrete inverse-depth levels), supervised by an exponentially weighted sequence loss: $\(\mathcal{L} = \sum_{t=1}^T \gamma^{T-t} |\hat{d}_{\mathrm{up}}^{(t)} - d^*|\)$ where \(d^*\) denotes the ground-truth inverse depth, and \(\gamma = 0.9\) prioritizes convergence in later iterations. The training schedule begins with 30 epochs of pre-training on OmniThings, followed by 1 epoch of student distillation, and concludes with 15 epochs of end-to-end fine-tuning on a mixed dataset of OmniHouse and Sunny. Optimization is conducted using AdamW with an initial distillation learning rate of \(5 \times 10^{-4}\) and weight decay of \(10^{-5}\).

Key Experimental Results

Main Results

Quantitative evaluations are conducted across the synthetic omnidirectional benchmarks OmniThings, OmniHouse, and three weather-conditioned driving scenarios (Sunny, Cloudy, Sunset). Metrics follow standard inverse-index error rates: percentage of pixels with index error greater than 1, 3, and 5 (\(>1, >3, >5\)), along with Mean Absolute Error (MAE) and Root Mean Square Error (RMS).

The following table summarizes performance across pre-trained and fine-tuned settings:

Dataset & Training Protocol Method >1 (%) ↓ >3 (%) ↓ >5 (%) ↓ MAE ↓ RMS ↓ Latency (ms) ↓
OmniThings (Pre-trained only) OmniMVS 47.72 15.12 8.91 2.40 5.27 289
RomniStereo64 17.77 7.52 5.00 1.22 3.90 161
MDP-Omni 18.37 7.09 4.59 1.20 3.79 117
Ours (OmniDS) 14.89 5.94 3.96 1.04 3.59 148
OmniHouse (Pre-trained only) RomniStereo64 10.52 4.05 2.69 0.74 1.73 161
MDP-Omni 17.13 7.20 4.68 1.16 2.62 117
Ours (OmniDS) 9.42 3.28 2.01 0.64 1.61 148
OmniThings (Fine-tuned) RomniStereo64-ft 29.84 16.21 11.28 2.26 5.60 161
MDP-Omni-ft 29.53 14.05 9.12 1.96 5.08 117
Ours-ft 25.14 11.72 7.67 1.70 4.69 148
Ours-ft (distilled) 26.22 13.69 9.51 2.11 5.49 102
OmniHouse (Fine-tuned) RomniStereo64-ft 5.28 2.22 1.51 0.42 1.14 161
MDP-Omni-ft 5.61 2.19 1.50 0.44 1.15 117
Ours-ft 3.92 1.17 0.64 0.28 0.89 148
Ours-ft (distilled) 3.41 1.16 0.73 0.28 0.89 102
Sunny (Fine-tuned) RomniStereo64-ft 4.61 1.78 1.10 0.32 1.43 161
MDP-Omni-ft 4.51 1.43 0.86 0.33 1.43 117
Ours-ft 3.75 1.39 0.86 0.28 1.32 148
Ours-ft (distilled) 3.85 1.43 0.87 0.27 1.30 102

Ablation Study

The components were ablated on the Sunny benchmark (trained for 30 epochs under identical protocols):

Table 1: Dual-stream representation and context fusion configurations (Sunny benchmark)

ID Sim. (CNN) Sim. (DINO) Sim. (MVC) Ctx. (CNN) Ctx. (DINO) Def. Attn. >1 (%) ↓ >3 (%) ↓ >5 (%) ↓ MAE ↓ RMS ↓
(a) - ✓ - - ✓ - 9.70 3.01 1.69 0.51 1.76
(b) ✓ - - ✓ - - 5.06 1.84 1.13 0.38 1.69
(c) ✓ - - - ✓ - 5.37 1.98 1.23 0.38 1.62
(d) ✓ - - ✓ ✓ - 4.69 1.69 1.07 0.35 1.55
(e) ✓ - ✓ ✓ ✓ - 4.60 1.74 1.09 0.35 1.63
(f) Full ✓ - ✓ ✓ ✓ ✓ 4.49 1.68 1.07 0.34 1.59

Table 2: Ablation of similarity and consensus volume components (Sunny benchmark)

ID Correlation (Corr) 2D Spatial Aggr. Multi-View Consensus (MVC) >1 (%) ↓ >3 (%) ↓ >5 (%) ↓ MAE ↓ RMS ↓
(a) ✓ - - 5.00 1.86 1.19 0.37 1.59
(b) ✓ ✓ - 4.67 1.68 1.06 0.34 1.52
(c) Full ✓ ✓ ✓ 4.49 1.68 1.07 0.34 1.59

Key Findings

  • Dual-stream synergy is essential: Isolating DINOv3 for both similarity and context (Config a) causes severe degradation (>1 index error increases to 9.70%), illustrating that low-resolution semantic tokens alone cannot resolve fine spatial parallax. Combining CNN and DINO contexts (Config d vs. b) drops >1 error from 5.06% to 4.69%, proving the value of global semantic priors in guiding ambiguous regions.
  • Deformable attention with distortion bias sharpens boundaries: Equipping context sampling with radial distortion bias and deformable attention (Config f vs. e) reduces >1 error to 4.49%, qualitatively eliminating boundary artifacts around poles, road signs, and thin railings.
  • MVC dominates fine-grained accuracy: As shown in Table 2, adding the Multi-View Consensus Volume (integrating 4-camera feature variance and group-wise correlation) reduces the fine-threshold >1 error from 4.67% to 4.49%, demonstrating that cross-camera agreement effectively suppresses false local minima.
  • Distillation delivers practical efficiency: The distilled MobileNetV2 model matches the full model's MAE of 0.28 on OmniHouse while cutting latency from 148 ms to 102 ms, confirming that high-level semantic context can be effectively compressed for robotic deployment.

Highlights & Insights

  • Optical Distortion Bias in Attention: Explicitly incorporating the fisheye lens's radial optical profile into cross-attention offsets guides feature gathering toward high-fidelity central rays, offering an elegant geometric trick for non-perspective vision tasks.
  • Decoupled Dynamic/Static Geometry Formulation: Using static 3D-regularized consensus volumes for multi-scale cost lookup while updating dynamic ERP contexts recurrently strikes an optimal balance between global candidate stability and occlusion-free boundary refinement.
  • Asymmetric Feature Distillation Strategy: Setting \(\lambda_{\mathrm{dino}}=10.0\) and \(\lambda_{\mathrm{cnn}}=1.0\) enables a compact MobileNet student to simultaneously absorb complex transformer semantic fields and sharp edge representations within a single training epoch.

Limitations & Future Work

  • Constrained Four-Camera Symmetry Assumption: The current framework relies on a fixed four-camera orthogonal layout (front/right/back/left); adapting the consensus volume and view weighting to arbitrary camera counts or asymmetrical baselines remains unaddressed.
  • Synthetic-to-Real Domain Gap: Training and evaluation are confined to synthetic benchmarks without modeling real-world sensor vibrations, mechanical calibration drift, lens glare, or cross-exposure inconsistencies.
  • Future Directions: Developing permutation-invariant cross-view attention consensus modules capable of processing unconstrained camera counts, combined with unsupervised real-world domain adaptation.
  • vs OmniMVS / CasOmniMVS: Traditional spherical sweeping models incur massive 3D convolution overhead (289 ms per frame); OmniDS replaces heavy 3D warping with a lightweight ConvGRU refinement pipeline and 3D MVC, cutting latency by over half while achieving superior precision.
  • vs RomniStereo: RomniStereo relies on pure CNN features with rigid ERP view-weighting, leaving thin boundaries prone to occlusion leakage; OmniDS introduces DINOv3 foundation priors and depth-guided dynamic re-sampling to preserve structural coherence.
  • vs MDP-Omni: MDP-Omni mitigates depth multi-modality via center-view sampling, sacrificing peripheral view cues; OmniDS's multi-view consensus volume directly measures multi-camera feature variance, preserving omnidirectional geometric consistency without view bias.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (The combination of distortion-aware deformable attention and cross-camera variance consensus is highly original)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive testing across 5 omnidirectional benchmarks with thorough modular ablations)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous problem formulation addressing fundamental visibility conflicts with clear prose and diagrams)
  • Value: ⭐⭐⭐⭐⭐ (Delivers state-of-the-art omnidirectional depth with a practical distilled variant ready for mobile robotics deployment)