Fisheye3R: Adapting Unified 3D Feed-Forward Foundation Models to Fisheye Lenses

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/android-xr/fisheye3r
Area: 3D Vision
Keywords: fisheye cameras, multi-view reconstruction, calibration tokens, masked attention, self-supervised adaptation

TL;DR

Fisheye3R equips frozen unified feed-forward 3D models with learnable calibration tokens and camera-type attention masks for native fisheye and mixed-camera inputs; adaptation using only unlabeled perspective images raises MapAnything's pose AUC on ScanNet++ fisheye testing from 0.519 to 0.766.

Background & Motivation

VGGT, π3, and MapAnything can predict camera poses and dense geometry in one forward pass from multiple uncalibrated images of a scene. This unified capability is learned from large perspective-image collections, which teach both scene knowledge and implicit expectations about local textures and spatial arrangements under perspective projection. Fisheye lenses change the mapping from three-dimensional directions to image pixels: the same object can appear curved or compressed near the image boundary. The scene itself may be familiar, yet the local patterns received by patch embedding are outside the pre-training distribution, leading subsequent cross-view inference to distorted point clouds and incorrect poses. This is particularly problematic for robotics, AR/VR, and vehicle-mounted mixed-camera systems, where fisheye lenses are chosen precisely for their wider environmental coverage.

Rectifying fisheye images into pinhole views does not solve the problem without trade-offs. A narrow viewport discards peripheral information; retaining a wide field of view introduces boundary stretching and resampling artifacts that can still fall outside the backbone's training distribution. Rectification also requires accurate calibration and additional processing, while fisheye images and their three-dimensional annotations are much scarcer than perspective data. Full fine-tuning may degrade existing perspective capability, whereas retraining a large model for both lens types incurs computational costs and camera-type data imbalance. The objective is therefore not a dedicated fisheye reconstructor, but continued access to existing scene knowledge under a different projection.

The authors move adaptation into latent space: images retain their fisheye projection, while a small collection of additional tokens adjusts internal backbone features. During training, existing perspective images can be distorted synthetically, and predictions can be mapped back to the original observation grid to reuse ground truth or the model's own predictions. This removes the requirement to begin adaptation by collecting labeled real fisheye data and provides an explicit interface for preserving the original model's behavior. Core Idea: freeze the geometric backbone, correct fisheye features using camera-selective layerwise calibration tokens, and reuse perspective supervision through distortion synthesis and inverse output transformation.

Method

Overall Architecture

The input consists of multiple RGB images of the same three-dimensional scene, captured entirely by fisheye cameras or by a mixture of perspective and fisheye cameras. The system identifies each frame's camera type from early frozen encoder features, then selectively activates calibration tokens in subsequent image encoding, frame-wise attention, and global attention. The original decoder still predicts poses and dense geometry; no separate fisheye reconstruction network is introduced. Distortion synthesis and inverse-transformation supervision are training branches only; inference does not require rectifying fisheye inputs into perspective images first.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view RGB"] --> B["Frozen early encoding"]
    B --> C["Camera-type masking"]
    C --> D["Layerwise calibration tokens"]
    D --> E["Original decoder<br/>Poses and dense geometry"]
    E -.->|Training only| F["Inverse-transformation supervision"]
    P["Perspective images"] -.->|Training only: synthesize distortion| A
    P -.->|Original predictions or ground truth| F
    Q["Real fisheye ground truth"] -.->|SL+: direct supervision| F

Dense geometry does not mean an identical output quantity for every backbone. VGGT predicts camera-coordinate depth and world-coordinate point maps; π3 predicts local point maps and transforms them into global point maps using the estimated poses. MapAnything predicts local ray directions and ray depths, combines them into camera coordinates, and then transforms them into global coordinates. Fisheye3R preserves these decoding interfaces, making it a shared adaptation strategy rather than a new geometric representation imposed on all backbones.

Key Designs

1. Camera-type masking: restrict calibration information to frames that need adaptation

Adding calibration tokens to every image also modifies perspective images that the original model already handles well. With fisheye-only training, these tokens specialize in correcting distortion and need not learn an identity operation for perspective inputs. The authors extract the class token at image-encoder layer \(L_0\) and classify the camera type using a linear classifier with sigmoid activation. Binary attention masks are then constructed for each frame and the entire sequence: fisheye frames can read calibration tokens, while perspective frames cannot. Classification uses early features before calibration, avoiding a decision about adaptation based on features that have already been adapted.

The masks control the direction of information flow in attention; they do not remove perspective frames or prohibit cross-camera matching. Equation (15) retains image-to-image connections and calibration tokens' access to image tokens, while conditioning image tokens' access to calibration tokens on the frame type. Images from different lenses can therefore still exchange scene information in global attention, allowing a forward-facing perspective camera to connect left and right fisheye views. For an entirely perspective sequence with correct classification, the frozen backbone and blocked calibration channel preserve the original computational path. In mixed sequences, perspective frames can still be affected indirectly by other adapted images, so this mechanism should not be interpreted as absolute per-frame output invariance.

2. Layerwise calibration tokens: correct features inside the frozen backbone instead of resampling images

Each adapted layer receives \(K\) learnable vectors that participate in attention alongside the original image tokens. In the image encoder, insertion begins only after the first \(L_0\) layers, preserving the preceding camera-type classification process. Calibration tokens are also inserted into frame-wise and global layers of the alternating-attention module, adapting both individual-frame representations and cross-view fusion. For a sequence with \(S\) frames and \(N\) image tokens per frame, frame-wise layers use \(K\) vectors to adapt each frame's \(N\) tokens separately. A global layer instead uses one set of \(K\) vectors jointly across the sequence's \(SN\) image tokens.

Different layers have their own calibration parameters rather than propagating the same updated auxiliary tokens throughout the network. Each layer discards its calibration-token outputs immediately after attention; only the adjusted image features proceed to the next layer. This localized injection allows each layer to learn a correction appropriate to its representation level while the original encoder, attention modules, and decoder remain frozen. Calibration tokens are not explicit lens intrinsics and do not directly produce a fisheye-to-pinhole pixel-coordinate table. They modify feature values through attention so that existing geometric decoding remains useful; stronger peripheral attention is supporting visualization provided by the authors, not an explicitly imposed boundary rule.

3. Inverse-transformation supervision: turn perspective data into usable fisheye training pairs

Training samples need not come from real fisheye cameras: the authors use the Kannala-Brandt model to synthesize fisheye distortion from perspective images. Let \(T\) denote this image distortion; the adapted model processes \(T(I^p)\) and produces dense predictions on the distorted grid. Before comparison with supervision, \(T^{-1}\) maps those dense predictions back to the original perspective observation locations, allowing loss computation at the ground-truth resolution. Camera extrinsics describe the camera's position and orientation in the scene and do not change when only the lens projection changes, so pose predictions do not receive this spatial inverse transformation. The important operation is aligning predictions with supervision, rather than repeatedly rectifying inputs or training against resampled ground truth.

Without geometric annotations, the frozen original model processes the undistorted perspective sequence and supplies pseudo-labels. The adaptation branch processes the synthetically distorted version of that sequence and is compared with the original branch after restoring the output grid. With perspective annotations, actual poses and dense geometry replace pseudo-labels; with real fisheye annotations, direct supervision on the native fisheye grid is also possible. These options share the adaptation parameters and the original backbone's task loss, without requiring an entirely new fisheye geometric objective. However, distorting a perspective image does not create scene content beyond its original field of view, so requiring no fisheye training data does not imply exposure to every ultra-wide-angle phenomenon.

A Worked Example

Consider the vehicle-mounted system in Figure 6: left and right fisheye cameras observe the sides, while a forward-facing perspective camera supplies cross-view connectivity. This is a walkthrough of the paper's scenario, not a new quantitative experiment, and it does not assume that the side views necessarily overlap directly. Each image independently passes through early encoding, after which the classifier identifies its camera type and constructs masks that allow or block calibration connections. Image features from the two fisheye cameras read calibration tokens in later layers; the forward-facing perspective frame does not read those tokens directly. Global attention still exchanges scene information across all three views, allowing the forward view to connect observations that would otherwise be difficult to combine. The decoder predicts per-frame poses and dense geometry, producing a unified reconstruction through shared scene inference rather than unrelated left and right point clouds. This inference process does not require users to supply precise fisheye intrinsics, but still depends on sufficient scene connectivity and correct camera-type classification.

Loss & Training

SSL uses unlabeled perspective RGB with original-model predictions as supervision; SL uses annotated perspective data; SL+ additionally incorporates annotated fisheye data. To avoid reproducing corrupted extracted equations, the following explicit shorthand summarizes Equations (11)–(12) on page 7, where \(U\) inversely transforms dense outputs only and leaves poses unchanged.

\[ \mathcal{L}_{\mathrm{SSL}}=\mathcal{L}\bigl(f(I^p),U(f(T(I^p);\phi))\bigr),\qquad \mathcal{L}_{\mathrm{SL}}=\mathcal{L}\bigl((E^p,D^p),U(f(T(I^p);\phi))\bigr). \]

Here \(f\) denotes the frozen backbone and \(\phi\) the calibration tokens; the branch without \(\phi\) is the original model, and \(\mathcal{L}\) is the corresponding backbone's task loss. For real fisheye samples in SL+, native outputs are compared directly with fisheye ground truth rather than transformed back to a perspective grid through \(U\). SSL and SL use 6 datasets: the perspective portion of ScanNet++, MegaDepth, BlendedMVS, TartanAir, MVS-Synth, and ParallelDomain-4D. SL+ additionally uses ASE and KITTI360 fisheye data; testing covers the fisheye portion of ScanNet++, ADT, and KITTI360. Consequently, SL+ results on KITTI360 should not be described as zero-shot transfer to a completely unseen dataset, even though training and testing can use different splits. The defaults are \(L_0=12\) and \(K=8\); the former supplies sufficiently deep classification features while reserving the remaining 12 encoder layers for calibration. The supplied main text does not specify a complete optimizer, learning-rate, training-epoch, or classifier-loss configuration, so no reproduction hyperparameters are invented here.

Key Experimental Results

Main Results

The following selection from Table 1 on page 9 retains clearly readable MapAnything rows and jointly examines pose, depth, point-map, and field-of-view behavior. Pose AUC integrates the angular accuracy curve up to \(30^\circ\), whereas FoV AUC uses a \(10^\circ\) threshold; these are different metrics. Depth Rel is absolute relative error, and CD is point-cloud Chamfer Distance; both are lower-is-better, and CD across datasets should not be compared without considering scene scale.

Test dataset	Config	Pose AUC ↑	Depth Rel ↓	Point-map CD ↓	FoV AUC ↑
ScanNet++	Original model	0.519	0.274	0.083	0.647
ScanNet++	SSL	0.766	0.190	0.054	0.875
ScanNet++	SL	0.730	0.195	0.054	0.853
ScanNet++	SL+	0.774	0.171	0.051	0.927
ADT	Original model	0.448	0.145	0.147	0.399
ADT	SSL	0.704	0.090	0.068	1.000
ADT	SL	0.691	0.091	0.068	1.000
ADT	SL+	0.753	0.087	0.058	1.000
KITTI360	Original model	0.428	0.258	1.301	0.259
KITTI360	SSL	0.540	0.156	1.015	0.291
KITTI360	SL	0.538	0.153	1.016	0.295
KITTI360	SL+	0.917	0.091	0.575	0.805

Ablation Study

Figure 7 on page 13 reports that camera classification saturates near \(L_0=12\), \(K=1\) already provides most of the gain, and point-map quality peaks around \(K=8\). The cache does not retain exact curve coordinates, so this graphical ablation is not converted into a numerical table with invented CD or classification accuracy values. Instead, the following projection-generalization analysis comes from Table 2 on page 14: π3 is tested on Stanford2D3DS renderings, while training synthesis uses only Kannala-Brandt (KB).

Test projection model	π3 CD ↓	π3 + Fisheye3R CD ↓	Reported improvement
KB (OOD parameters)	0.230	0.116	49.7%
Fisheye624	0.228	0.117	48.5%
MEI	0.264	0.095	64.1%
Equidistant	0.219	0.107	51.1%
Stereographic	0.211	0.088	58.1%
Equiangular	0.241	0.116	52.0%
Orthographic	0.375	0.182	51.5%

Improvement percentages are preserved from the source table; they need not be exactly reproducible from CD values displayed to three decimal places, and the paper does not provide unrounded values.

Key Findings

• Real fisheye annotations are not a prerequisite for effective adaptation: SSL reduces MapAnything's ScanNet++ CD from 0.083 to 0.054.
• More supervision does not improve every metric monotonically: SSL pose AUC is 0.766 on that test, exceeding SL's 0.730.
• Real fisheye supervision has substantial value outdoors: KITTI360 MapAnything pose AUC rises from 0.538 with SL to 0.917 with SL+, but this is not a purely cross-dataset zero-shot comparison.
• Page 11 reports improvements exceeding 50% in 26/135 metrics for SSL, 37/135 for SL, and 77/135 for SL+; these are counts of heterogeneous metrics, not a unified average accuracy.

Highlights & Insights

• The central distinction is that projection-induced representation shift and missing scene knowledge need not be the same problem. Improvements across geometric tasks with a frozen backbone show that relearning all geometry is not the only option.
• Masking turns perspective compatibility from an incidental training expectation into a controlled information pathway. It preserves cross-camera fusion instead of assigning the two camera types to isolated models.
• Distortion synthesis and inverse output transformation make existing perspective data reusable for adaptation. SSL also obtains supervision from the input distribution the foundation model handles best rather than asking it to pseudo-label unfamiliar fisheye images directly.
• Peripheral patches attend more strongly to calibration tokens in Figure 8, consistent with the distortion pattern. However, attention maps and t-SNE support a mechanism interpretation rather than independently proving causality or complete distribution alignment.

Limitations & Future Work

• The authors explicitly note that the method depends on transformer token interfaces and cannot be transferred unchanged to arbitrary convolutional backbones.
• SSL pseudo-label quality is limited by the original model, so teacher geometry errors can persist; ground truth can help when available, but annotations are not guaranteed for every task.
• Reader assessment: camera-type errors activate the wrong adaptation pathway, motivating tests on mild distortion, unusual crops, and other lens types; the current plots do not replace failure-rate measurements for these cases.
• The panoramic extension is not a zero-shot demonstration of the existing fisheye tokens: Figure 9 trains on Matterport3D and tests on Stanford2D3DS, demonstrating adaptation potential for another projection.
• The authors emphasize low additional overhead, but the supplied main text lacks a complete parameter-count, latency, and training-compute comparison, so no specific speedup factor can be claimed.
• Source boundary: several equations and some π3/VGGT rows in Table 1 are merged in the cache; ambiguous values are not quoted, training equations use explicitly identified shorthand, and graphical findings are not presented as exact coordinates.

Related Work & Insights

• vs VGGT, π3, and MapAnything: these models supply unified feed-forward geometry, while Fisheye3R addresses the mismatch between perspective pre-training and fisheye inputs as an additional adaptation layer rather than a replacement backbone.
• vs monocular-depth calibration tokens: prior work cited by the paper already used calibration tokens for monocular depth; this work extends adaptation to multi-view frame-wise/global interactions, multiple geometric outputs, and mixed-camera compatibility, so the token concept itself should not be treated as entirely new.
• vs explicit rectification and traditional SfM/MVS: traditional methods handle lenses explicitly in projection models and geometric solvers; this work lets frozen learned representations receive distorted images natively, which does not imply that camera geometry or calibration errors can always be ignored.
• Research direction: compare binary camera gating with continuous distortion-aware gating, measuring boundary cases while retaining perspective regression tests; this is a reader-proposed question, not a validated result of the paper.

Rating

These are the note author's subjective assessments, not conference review scores. - Novelty: 4/5. Calibration tokens have precedents, but mixed-camera control and unified multi-view adaptation provide a clear extension. - Experimental Thoroughness: 4/5. Evaluation spans 3 backbones, 3 main test datasets, and 3 supervision schemes, while fuller efficiency and camera-misclassification analyses remain needed. - Writing Quality: 4/5. The relationship between the problem, adaptation mechanism, and data constraints is clear; extraction-corrupted equations are not counted as a writing flaw in the paper. - Value: 4/5. Relevant to 3D perception systems that need fisheye support while retaining existing perspective capabilities.