Skip to content

From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Conference: ECCV2026
Paper: ECCV 2026
Code: https://github.com/Suchisrit/DEX
Area: 3D Vision
Keywords: fisheye camera / monocular depth estimation / open-vocabulary segmentation / distortion adaptation / latent-space alignment

TL;DR

The paper introduces Distortion Extenders (DEX), a set of lightweight learnable modulators inserted into every layer of a frozen backbone, which pull the feature distribution of fisheye images back toward the perspective feature distribution so that depth estimators and open-vocabulary segmentation models pretrained only on perspective data transfer zero-shot to fisheye cameras; training needs neither ground truth nor real fisheye images, and inference needs neither calibration nor re-projection. DEX improves RMSE/Ξ΄1 and mIoU on three real fisheye datasets (ScanNet++, KITTI-360, WoodScape), and its activations can be decoded back into KB distortion coefficients for camera calibration.

Background & Motivation

Today's vision foundation models are trained on hundreds of millions of internet images, the overwhelming majority of which were captured by ordinary perspective cameras, whose imaging geometry is well approximated by a simple pinhole projection. The projective bias of that camera model is therefore "baked into" the weights: these models generalize across wildly different 3D scenes yet remain highly sensitive to the projection model itself. Real deployments (XR, robotic manipulation, autonomous navigation) are mixed-camera systems that inevitably include wide field-of-view (wide FoV) lenses such as fisheye; their projection is strongly non-linear, objects near the image border are increasingly warped, and the local pixel arrangement shifts systematically relative to perspective images. The encoded features shift with it, producing distorted 3D reconstructions and degraded recognition β€” a textbook covariate shift rather than a matter of model capacity.

The two established remedies are both unsatisfying. The first "undoes" the distortion: rectify with known intrinsics, or re-project into a canonical representation such as equirectangular (ERP), tangent, spherical, or cube-map projection before feeding the network. This preserves compatibility with perspective-pretrained models, at the cost of multiple resampling steps, a hard dependence on test-time calibration, and representation-specific artifacts β€” vertical stretching and angular compression near the poles for ERP, radial stretching near the edges for tangent projection, discontinuous seams between cube faces β€” all of which are then amplified by convolutions, tokenization, and patch embedding. The second remedy trains dedicated models or fine-tunes large pretrained ones for a specific camera; fisheye data are far scarcer than perspective data, and fine-tuning risks parameter drift, trading the model's original generalization across scenes and previously learned cameras for fidelity on one lens, while a mixed-camera platform must maintain a separate estimator per lens.

More fundamentally, both routes push camera-specific inductive bias back into the design: specialized operators for one projection, or task-specific losses for one task, which constrains the end user's freedom in choosing both architecture and task. This paper's angle is to change the hypothesis instead. Since a frozen pretrained model can already infer 3D scene properties faithfully from perspective images, the errors on fisheye images come from a shift in the values of latent embeddings β€” distortion displaces and rearranges pixels, so local operations over them yield different numbers. If those values can be modulated back toward the distribution that produced high-fidelity estimates, the model's capability extends to fisheye cameras without touching projective geometry and without retraining the backbone. Core idea: freeze the backbone and insert a set of learnable "distortion extenders" after every layer, modulating latent embeddings by a convex combination weighted by each feature's own affinity, and train them self-supervised by inverse geometric alignment between a perspective image and its synthetically distorted fisheye copy β€” one mechanism that serves both CNNs and Transformers, both depth regression and open-vocabulary segmentation, with no calibration at test time.

Method

Overall Architecture

Formally, a model pretrained on perspective images consists of an image encoder \(f(\cdot)\) and a task decoder \(g(\cdot)\); both are frozen here. Inference for a task is \(\hat{y}=g(f(I))\), where \(C=1\) for regression tasks such as monocular depth (the estimate being a 2.5D range or depth map) and \(C\) varies with the number of classes or text labels for classification tasks such as open-vocabulary segmentation (the estimate being affinities between image and text features). DEX must translate the latent embeddings of a fisheye image \(I_f\) into the perspective embedding distribution without changing \(f\) or \(g\). That requires two steps: determining how strong the distortion is and what it does to the latent embeddings, and then determining the modulation needed to correct for it. DEX therefore has two components: a set of \(M\) Extender modulation parameters \(\mathbf{E}\in\mathbb{R}^{M\times D}\) that act as discrete "centers" in latent space, and a "soft" selection mechanism parameterized by a weight matrix \(\mathbf{W}\) that lets every feature vector decide for itself which convex combination of centers should correct it. The parameters are organized per layer as \(\theta=\{\mathbf{E}^{(l)},\mathbf{W}^{(l)}\}_{l=1}^{L}\) and inserted after the \(L\) blocks of the encoder.

Training is self-supervised. From a calibrated perspective image, a fisheye counterpart is synthesized with a known KB distortion model; the perspective image goes through the frozen backbone to produce a reference output, the fisheye image goes through the same backbone augmented with DEX to produce the output to be aligned, and the fisheye branch is mapped back into the perspective reference frame with the inverse transformation \(T^{-1}\) so the two can be compared directly. No ground truth and no real fisheye image are involved. At test time the procedure is the opposite of elaborate: the fisheye image is fed straight into the augmented model, with no calibration, no re-projection, and no additional inference-time geometry. The only overhead is a few megabytes of parameters and milliseconds of latency.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["calibrated perspective image I"] --> B["synthesized fisheye image<br/>KB distortion model T"]
    A --> C["frozen backbone f<br/>perspective reference output"]
    B --> D["Extender soft selection<br/>convex combination per distortion level"]
    C --> E["inverse geometric alignment<br/>minimize loss after T⁻¹"]
    D --> E
    E -->|depth task| F["spherical range parameterization"]
    E -->|segmentation task| G["latent-space segmentation alignment"]

Key Designs

1. Extender soft selection: shifting latent features by a convex combination weighted per distortion level

Re-projection, specialized convolution kernels, and camera-specific inductive bias all share an implicit assumption: that the distortion is uniform across the image. Fisheye distortion is precisely the opposite β€” radial and non-uniform, with the image center almost equivalent to a perspective projection and the deviation growing toward the border. A single global transform therefore over-corrects the center and under-corrects the periphery. DEX pushes the correction down to each individual feature vector. At the output \(\mathbf{X}^{(l)}\in\mathbb{R}^{(h\times w)\times D}\) of encoder block \(l\) β€” a token for ViTs, a flattened feature vector for CNNs, which is exactly where its architecture-agnosticism comes from β€” a query is generated from \(\mathbf{E}\) and \(\mathbf{W}\), each feature's affinity to that query is computed by scaled dot-product attention, and the Extenders are then combined convexly with those affinities as weights and added back:

\[\mathbf{S}=\mathbf{E}\mathbf{W},\qquad \mathbf{A}=\mathrm{softmax}\!\left(\frac{\mathbf{X}\mathbf{S}^{\top}}{\sqrt{D}}\right),\qquad \mathbf{X}\leftarrow\mathbf{X}\mathbf{A}\mathbf{E}\]

The softmax makes the weights non-negative and sum-to-one, so \(\mathbf{X}\mathbf{A}\mathbf{E}\) is a convex combination of Extenders: \(\mathbf{E}\) behaves like a set of "centers" or codebook entries in latent space, and every feature vector picks where it falls among them according to its own affinity. Because distortion varies continuously while the combination weights are continuous, these centers bound a region that covers a continuous range of distortion strengths β€” the more distorted the input, the further the selected combination moves from the undistorted end. The modulated embeddings \(\mathbf{X}^{(l)}+\mathbf{A}\mathbf{E}\) are fed to the next layer and to any skip connections (⚠️ equation (2) is corrupted in the cached PDF extraction; the form here follows the paper's prose β€” refer to the original paper). To curb over-parameterization, \(\mathbf{W}\) is represented by the low-rank factorization \(\mathbf{W}_2\mathbf{W}_1^{\top}\) with \(\mathbf{W}_1,\mathbf{W}_2\in\mathbb{R}^{d\times D}\) and \(d\ll D\), motivated by a concrete observation: the covariate shift originates in changes to calibration, and calibration space is intrinsically low-dimensional, so the transform carrying it should be low-rank too. Compared with Calibration Tokens, DEX neither appends tokens to the residual stream nor manually removes them layer by layer, and because it acts on the set of feature vectors it works for CNNs as well; compared with LoRA, which learns a weight matrix projecting the input directly into an additive bias, DEX learns modulator parameters plus a selection projection so that the input features themselves determine the convex-combination weights.

2. Inverse geometric alignment: training self-supervised from perspective images alone

The scarcity of fisheye data is the hardest constraint on this line of work β€” there are neither enough real fisheye images nor, more importantly, matching ground-truth depth. The paper sidesteps the constraint by turning the domain shift itself into the supervision signal. Given a calibrated perspective image \(I\), a forward transformation \(T\) under the KB distortion model yields a synthetic fisheye image \(I_f=T(I)\); the perspective branch output \(\hat{y}=g(f(I))\) serves as reference, and the fisheye branch output \(\hat{y}_f=g(f_\theta(I_f))\) is what must be aligned. Since the two differ only by a known geometric transform, \(\hat{y}_f\) can be mapped back into the perspective reference frame by the inverse transformation \(T^{-1}\) before minimizing the alignment loss, i.e. optimizing

\[\arg\min_{\theta}\ \frac{1}{N}\sum_{n=1}^{N}\mathcal{L}\big(\hat{y}^{(n)},\,T^{-1}\hat{y}_f^{(n)}\big)\]

The elegance is that the supervision signal and the quantity being supervised differ by a known and invertible geometric transform. The model is therefore forced to learn how to make distorted features produce predictions consistent with perspective ones, rather than fitting the imaging idiosyncrasies of one camera. Training needs neither ground truth nor a single real fisheye image; the only cost is the resampling involved in synthesizing the distortion. This also explains a mildly counter-intuitive result later on: one and the same set of Extenders serves both an indoor domain (ScanNet++) and an outdoor one (KITTI-360) with different distortion parameters and scene types. What it borrows is the frozen backbone's own generalization; DEX merely realigns that capability to wide-FoV input. The training paradigm follows AugUndo's self-supervised recipe, and freezing the backbone sidesteps parameter drift by construction, unlike direct fine-tuning.

3. Spherical range parameterization: replacing Cartesian depth with distance along the viewing ray

Perspective models output z-depth, the forward distance in Cartesian coordinates. As the FoV widens, rays near the image border become nearly perpendicular to the optical axis and the same physical distance is pushed to an extreme value in z β€” precisely the radial attenuation that perspective-pretrained models exhibit at fisheye borders. Outdoor data such as KITTI-360 have a wider FoV and stronger distortion than indoor data, so Cartesian parameterization also forces the model to relearn the scale relation under every distortion level. DEX therefore changes the output parameterization on the depth branch as well: each pixel of the perspective depth map is converted to a spherical range, and the fisheye branch regresses \(R\) directly. For each pixel, combining the normalized image-plane coordinates with the depth gives

\[R=\left|\,z\cdot\begin{bmatrix}X/z\\Y/z\\1\end{bmatrix}\right|=\sqrt{X^{2}+Y^{2}+z^{2}}\]

Since DEX is already learning how to move an output from one coordinate system to another, the Cartesian-to-spherical re-projection is just one more learnable objective sharing the same Extenders. \(R\) is measured along the viewing ray and is independent of where the pixel falls in the image, so it does not blow up at the border the way z does. The ablation quantifies this sharply: removing the \(R\) extension raises ScanNet++ RMSE from 0.200 to 0.249 and drops Ξ΄1 from 0.872 to 0.792; outdoors on KITTI-360 it collapses from 1.663 to 7.024 and Ξ΄1 from 0.842 to 0.271 β€” the largest drop of any ablation, confirming that coordinate parameterization matters more as the FoV grows.

4. Latent-space segmentation alignment: supervising on image embeddings to bypass the text modality

Open-vocabulary segmentation architectures typically encode the image \(f_i(I)\) and a set of text labels \(f_l(t)\) separately, then compare them by inner product to assign a class to each pixel. Putting the alignment loss on the network's final output β€” after multiplication with the language modality β€” would dilute the supervision with the text side's uncertainty, whereas distortion only corrupts the image encoder, which is independent of the text encoder. DEX therefore moves supervision forward to the last-layer image embedding and compares the perspective image embedding with the aligned fisheye image embedding,

\[\mathcal{L}=\log\!\Big(\big|f_i^{(L)}(I)-T^{-1}\circ f_{i,\theta}^{(L)}(I_f)\big|+1\Big)\]

(⚠️ this equation is corrupted in the cached PDF extraction; whether the right-hand side explicitly contains \(T^{-1}\) should be verified against the original paper.) The text branch stays frozen and the inner product at inference proceeds as before. The change looks small but it is the origin of DEX's task-agnosticism: depth and segmentation use the same objective form and differ only in which tensor gets aligned β€” one in output space, one in latent space. It is also because supervision sits in the latent layer that the compliance of fisheye features with perspective features can be observed directly with t-SNE.

Loss & Training

The objective is unified. The depth branch aligns in output space with \(\mathcal{L}(\hat{R},\hat{R}_f)=\log(|\hat{R}-\hat{R}_f|+1)\), where the log compresses large errors; the segmentation branch aligns in latent space with the equation above. All training data are calibrated perspective images β€” NYUv2 (general indoor), VOID (perspective office, classroom, and stairwell scenes), IRS (synthetic rendered indoor), Hypersim (photorealistic indoor), and Waymo (urban driving) β€” mixed into roughly 200K training examples (80K for VNL), with fisheye counterparts synthesized by the KB model. One set of Extender parameters is shared between the indoor and outdoor evaluation domains. Inference overhead is minimal: +2.8 MB of memory and +0.3 ms on UniDepthV2, +6.1 MB and +2.21 ms on LSeg, negligible against the backbones' 0.7 GB and 2.9 GB. As a byproduct, DEX activations can be decoded back into distortion coefficients; the linear probe \(\mathbf{W}_D\) and the cross-layer weighting \(\mathbf{W}_L\) used for this are optimized on the same training data.

Key Experimental Results

Main Results

Table 1: Monocular depth estimation (MDE) zero-shot on real fisheye datasets (RMSE↓ / Ξ΄1↑). The same set of Extenders is used for indoor and outdoor.

Test set Model Train data Base +LoRA +Calibration Tokens +DEX
ScanNet++ [68] MiDaS [39] Mix 1.4M / 200K 0.672 / 0.479 0.561 / 0.599 0.413 / 0.688 0.397 / 0.702
ScanNet++ [68] DepthAnything [65] Mix 63.5M / 200K 0.705 / 0.454 0.706 / 0.445 0.479 / 0.667 0.414 / 0.708
ScanNet++ [68] UniDepthV2 [36] Mix 16M / 200K 0.329 / 0.671 0.235 / 0.854 0.223 / 0.841 0.200 / 0.872
ScanNet++ [68] VNL [69] (CNN) NYUD-V2 29K / Mix 80K 0.680 / 0.521 β€” β€” 0.372 / 0.704
ScanNet++ [68] UniK3D [35] (competitor) Mix 12M / 200K 0.223 / 0.825 0.218 / 0.835 β€” β€”
ScanNet++ [68] DepthAnyCamera [13] (competitor) Indoor 670K 0.390 / 0.852 β€” β€” β€”
KITTI-360 [26] MiDaS [39] Mix 1.4M / 200K 6.111 / 0.312 2.479 / 0.601 2.588 / 0.699 2.451 / 0.657
KITTI-360 [26] DepthAnything [65] Mix 63.5M / 200K 6.484 / 0.318 2.224 / 0.676 2.897 / 0.550 2.022 / 0.745
KITTI-360 [26] UniDepthV2 [36] Mix 16M / 200K 7.093 / 0.262 1.916 / 0.771 1.788 / 0.763 1.663 / 0.842
KITTI-360 [26] VNL [69] (CNN) KITTI 24K / Mix 80K 7.719 / 0.245 β€” β€” 2.222 / 0.605
KITTI-360 [26] UniK3D [35] (competitor) Mix 12M / 200K 2.969 / 0.812 2.802 / 0.818 β€” β€”
KITTI-360 [26] DepthAnyCamera [13] (competitor) Outdoor 130K 3.641 / 0.789 β€” β€” β€”

Table 2: Open-vocabulary segmentation and multitask evaluation. LSeg/SED are evaluated on WoodScape (mIoU↑ / weighted IoU↑); PanopticDepth is a multitask framework for depth estimation and depth-aware panoptic segmentation.

Test set Model Config Train data mIoU ↑ weighted IoU ↑
WoodScape [70] LSeg [22] Baseline Mix 500K 0.305 0.819
WoodScape [70] LSeg [22] w/ Calibration Tokens Mix 50K 0.321 0.823
WoodScape [70] LSeg [22] w/ Distortion Extenders Mix 50K 0.362 0.838
WoodScape [70] SED [62] (CNN) Baseline Mix 120K 0.439 0.829
WoodScape [70] SED [62] (CNN) w/ Distortion Extenders Mix 50K 0.449 0.832
WoodScape [70] / KITTI-360 [26] PanopticDepth [11] Baseline β€” 0.160 (mIoU) / RMSE 7.273 0.300
WoodScape [70] / KITTI-360 [26] PanopticDepth [11] w/ Distortion Extenders β€” 0.282 (mIoU) / RMSE 3.583 0.833

Ablation Study

Table 3: Ablations with UniDepthV2 as the backbone (RMSE↓ / Ξ΄1↑). Removing the low-rank decomposition and removing the spherical \(R\) extension each correspond to one architectural decision.

Dataset Config RMSE ↓ Ξ΄1 ↑ Note
ScanNet++ [68] Distortion Extenders 0.200 0.872 full model
ScanNet++ [68] w/o \(\hat{R}\) Extension 0.249 0.792 reverts to Cartesian z-depth; a clear drop even indoors
ScanNet++ [68] w/o Low-Rank Decomposition 0.216 0.849 over-parameterization causes a mild degradation
KITTI-360 [26] Distortion Extenders 1.663 0.842 full model
KITTI-360 [26] w/o \(\hat{R}\) Extension 7.024 0.271 nearly fails under wide-FoV outdoor distortion
KITTI-360 [26] w/o Low-Rank Decomposition 1.777 0.800 smaller drop than the \(R\) extension

Table 4: Decoding DEX activations into KB distortion coefficients (ScanNet++, lower is better).

Metric k1 k2 k3 k4 Average
MSE ↓ 2.04Γ—10⁻⁢ 1.53Γ—10⁻⁢ 3.53Γ—10⁻⁷ 8.87Γ—10⁻⁹ 9.81Γ—10⁻⁷
MAE ↓ 1.17Γ—10⁻³ 9.88Γ—10⁻⁴ 5.27Γ—10⁻⁴ 7.05Γ—10⁻⁡ 6.88Γ—10⁻⁴
MAPE (%) ↓ 3.91 14.30 41.86 78.61 34.67

Key Findings

  • Spherical \(R\) parameterization is the single largest contributor, and its importance grows with the FoV: 0.200 β†’ 0.249 on ScanNet++ versus 1.663 β†’ 7.024 on KITTI-360. A gap this large between indoor and outdoor indicates that the numerical degradation of Cartesian z-depth at wide-FoV borders is the dominant failure mode, not a minor detail.
  • The low-rank decomposition buys robustness against over-parameterization at a modest magnitude: 0.200 β†’ 0.216 on ScanNet++, 1.663 β†’ 1.777 on KITTI-360 β€” a refinement next to the \(R\) extension rather than a headline.
  • Average improvements: on indoor MDE, DEX improves RMSE by 19% and Ξ΄1 by 15% on average over LoRA and Calibration Tokens; UniDepthV2 with DEX improves over UniK3D by 8% averaged across metrics. Outdoors on KITTI-360 the average gain is 11% RMSE and 11% Ξ΄1 over both. On segmentation, DEX improves mIoU over Calibration Tokens by about 13%.
  • One honest counterexample: on the KITTI-360 MiDaS row, DEX has the lower RMSE (2.451 vs 2.588) but a slightly lower Ξ΄1 than Calibration Tokens (0.657 vs 0.699), so "uniformly better" does not hold on every individual metric; the advantage is stable only when the three backbones are viewed together.
  • The UniK3D comparison needs a caveat: UniK3D was itself trained on large-scale fisheye data (ASE, aiMotive, HOI4D, DL3DV, roughly 3.263M examples), whereas DEX used only about 200K mixed examples; applying LoRA on top of UniK3D yields only a marginal change (2.969 β†’ 2.802), which the authors take as evidence that its performance has largely saturated and that further adaptation with a much smaller set makes little difference. The training-set scales of the two are not directly comparable.
  • Broad pretraining data is a precondition: since supervision comes from the frozen backbone's perspective outputs, DEX is upper-bounded by the backbone's quality. The authors note that FMDEs and open-vocabulary segmentation models have been trained on tens to hundreds of millions of images, which lets training data be selected casually; for smaller-scale models, dataset selection requires care.
  • The network really uses the Extenders rather than treating them as noise: DEX activations decode into KB coefficients with a simple linear probe, and the accuracy degrades strictly from \(k_1\) to \(k_4\) (MAPE 3.91% β†’ 14.30% β†’ 41.86% β†’ 78.61%). This matches the visual effect of the coefficients β€” a small change in \(k_1\) affects the image far more than a change in \(k_4\), so later coefficients are intrinsically less discriminative and carry wider uncertainty. The authors also verify this visually: perturbing all coefficients by the average prediction error produces two synthesized fisheye images that are nearly indistinguishable to the eye, showing that the later coefficients are simply hard to recover at high fidelity.

Highlights & Insights

  • Reframing a geometry problem as a distribution-alignment problem: the "aha" of the paper is that it never models projective geometry and never re-projects. Its claim is that the consequence of distortion is already visible as latent embeddings drifting away from the distribution seen during pretraining, so camera adaptation becomes a pure latent-space transport problem. That single change of view eliminates the calibration dependence, the resampling artifacts, and the camera-specific inductive bias at once.
  • A known and invertible geometric transform is free supervision: the perspective-to-synthetic-fisheye distortion is invertible, so the perspective branch's output becomes the label for the fisheye branch and no ground truth is needed. The recipe (frozen backbone + synthetic domain shift + inverse-transform alignment + lightweight modulators) is generic and transfers to any setting where the input is corrupted by a known transform but the task output should be unchanged β€” motion blur, rolling shutter, color/exposure shift, or sensor-domain gaps.
  • Task-agnosticism comes from where the loss is written: rather than putting the segmentation loss on the logits, it is placed on the last encoder-layer embedding, bypassing the dilution from the text modality. The same "align before the two modalities diverge" trick transfers to any image-text dual-encoder adaptation problem, such as retrieval, referring segmentation, or open-vocabulary detection.
  • Adaptor activations as a calibration readout: DEX's convex-combination weights are a per-layer, per-sample characterization of distortion strength, and a linear probe reads KB coefficients out of them without any extra calibration network. That is practically useful for online calibration or for monitoring calibration drift in deployment.
  • Plug-in and extremely light: 2.8 MB / +0.3 ms for depth and 6.1 MB / +2.21 ms for segmentation let DEX ride on top of an existing model instead of replacing the deployment stack, which is far more realistic for mixed-camera systems than training one unified model.

Limitations & Future Work

  • The performance ceiling is set by the backbone (acknowledged by the authors): supervision is the frozen backbone's perspective output, so if the backbone is wrong, DEX learns the wrong thing. The authors argue the training set can be chosen casually because backbones are trained on tens to hundreds of millions of images, while conceding that small-scale models demand careful dataset selection. A direct consequence is that DEX cannot correct systematic errors the backbone already makes on perspective images.
  • The synthetic-to-real gap is never quantified: training uses KB-synthesized fisheye images while evaluation lands on real fisheye data, yet no sensitivity analysis is given for the mismatch (e.g. how performance degrades with a non-KB distortion model at training time or with a synthesized parameter distribution that does not match the test lens).
  • No "direct fine-tuning" baseline: the comparisons are LoRA, Calibration Tokens, and UniK3D; fine-tuning the backbone directly on the same 200K synthetic fisheye examples is missing. Since DEX freezes the backbone, that baseline is exactly what would isolate where the benefit of freezing comes from.
  • The calibration byproduct is weak on the later coefficients: MAPE reaches 41.86% for \(k_3\) and 78.61% for \(k_4\). The authors explain this by visual insensitivity, but it means "DEX can serve directly as a calibrator" holds only for the first two coefficients; a real calibration use would need extra constraints (regress only \(k_1,k_2\), or combine with calibration-board priors).
  • Improvement directions: (1) ablate the number of Extenders \(M\) and the rank \(d\) explicitly and study how they relate to the distortion-strength distribution; (2) add a small amount of weakly supervised real fisheye data to close the synthetic-to-real gap; (3) extend the framework to other fisheye tasks (optical flow, 3D detection, VLA) to test whether latent alignment is genuinely task-agnostic.
  • vs Calibration Tokens [10] (the closest concurrent work): it also extends a perspective-pretrained FMDE to fisheye, but by appending trainable tokens to Transformer blocks. Four differences: (1) DEX is architecture-agnostic and works for CNNs as well as Transformers, whereas the token approach is tied to Transformers; (2) Calibration Tokens require manually removing the appended tokens after each layer, while DEX appends and discards nothing and instead uses affinity to determine the convex-combination weights automatically; (3) Calibration Tokens use a depth-specific loss, whereas DEX's loss applies to both depth and segmentation; (4) a minor difference is that Calibration Tokens output z-depth (Cartesian) while DEX outputs range (Euclidean). Tables 1 and 2 show DEX ahead on both task families.
  • vs LoRA [16]: LoRA learns a weight matrix that projects the input directly into an additive bias; DEX learns a set of modulators plus a selection projection so the input features themselves decide the convex-combination weights. In Table 1, LoRA is actually worse than the base model on DepthAnything (0.706 vs 0.705), suggesting that generic parameter-efficient fine-tuning is not automatically suited to distortion adaptation β€” the shift caused by distortion is not of the low-rank weight-perturbation form.
  • vs UniK3D [35] / DepthAnyCamera [13] / FoVA-Depth [27]: this line re-projects the input or an internal representation into a canonical space (spherical internal representation, ERP, cube map) and trains on mixed-camera data. DEX does the opposite: no re-projection and no spherical operators, only latent-space alignment. The cost is dependence on the backbone; the benefit is an extremely low deployment cost and zero assumptions about the distorted input.
  • vs AugUndo [61]: DEX's self-supervised training paradigm is built directly on AugUndo (synthetic augmentation plus its inverse transform as supervision), but AugUndo targets augmentation scaling for depth completion/estimation, whereas DEX treats the domain shift itself as the object to be removed and introduces plug-in modulators rather than fine-tuning the backbone.
  • Takeaway: the trio of frozen backbone, lightweight latent modulation, and inverse-transform self-supervision is essentially a general template for any adaptation problem in which the input distribution is changed by a known mechanism while the task semantics stay fixed. That it transfers is already evidenced twice within this single paper β€” depth aligns in output space, segmentation in latent space β€” which makes it a reasonable default starting point for future multi-sensor and multimodal adaptation.

Rating

  • Novelty: β­β­β­β­β˜† Reframing fisheye adaptation from geometric rectification to latent distribution alignment, with an architecture- and task-agnostic lightweight implementation and a calibration byproduct, is a clean angle; the components themselves (low-rank modulation plus attention-style convex combination) are related to LoRA and attention, making this a combinatorial contribution.
  • Experimental Thoroughness: β­β­β­β­β˜† Three real fisheye datasets, five backbones (CNN and ViT), two task families, plus t-SNE, low-rank and coordinate-parameterization ablations, and an overhead comparison; the missing pieces are a direct fine-tuning baseline and a synthetic-to-real gap analysis.
  • Writing Quality: β­β­β­β­β˜† The motivation chain is clear and the four-point contrast with Calibration Tokens is concrete; however, several equations are corrupted in the source's typesetting and Table 1 is densely packed, so the reader must align rows manually.
  • Value: β­β­β­β­β˜† Low deployment cost (megabyte-scale parameters, millisecond latency), plug-and-play, no backbone modification β€” an extremely low barrier to retrofitting existing mixed-camera systems and foundation-model pipelines, with a usable calibration byproduct on the side, giving it more practical value than comparable adaptation work.