Skip to content

Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations

Conference: ECCV 2026
Paper: ECCV Official Portal
Code: https://github.com/palmdong/RobSelf
Area: Image Restoration
Keywords: Cross-modal super-resolution, Self-supervised super-resolution, Spatial misalignment, Guided image restoration, Content-aware filtering

TL;DR

RobSelf tackles the dual challenges of spatial misalignment and ground-truth absence in real-world multi-modal imaging through an online self-supervised model that jointly optimizes a weakly-supervised feature translator and a content-aware reference filter, delivering high-resolution and high-fidelity predictions without requiring training data, ground-truth supervision, or pre-alignment.

Background & Motivation

Multi-modal imaging modalities such as RGB, depth, and near-infrared (NIR) capture complementary structural and semantic information of physical scenes. Owing to physical sensor limitations and hardware constraints, non-visible modalities typically suffer from considerably lower spatial resolution than RGB cameras. This discrepancy has motivated cross-modal super-resolution (SR), which enhances a low-resolution (LR) source image by leveraging fine structural cues from a high-resolution (HR) guide image of another modality. While supervised methods require massive paired datasets and ground-truth HR supervision that are labor-intensive to collect in specialized domains, self-supervised methods optimize online per test pair without external training data, exhibiting much stronger practical generalization.

Nonetheless, almost all existing cross-modal SR methods rely on the strong assumption that the source and guide images are strictly pixel-aligned in space. In real-world multi-sensor systems, spatial misalignments are practically unavoidable due to intrinsic sensor geometry (such as lens distortions, field-of-view differences, and baseline displacements) as well as extrinsic platform perturbations (viewpoint variations and object motions). Conventional two-stage pipelines perform pre-alignment using off-the-shelf registration networks prior to super-resolution. However, these pre-alignment models frequently break down when coping with substantial resolution discrepancies and severe cross-modal domain shifts, producing boundary distortion and ghosting artifacts. Meanwhile, existing end-to-end alignment-aware SR methods either overfit to synthetic misalignments or use off-the-shelf deformation layers that overlook cross-modal dependencies.

Addressing the dual absence of high-resolution target supervision and spatial alignment ground truth, this paper reformulates cross-modal and cross-resolution alignment as an intrinsic weakly-supervised feature translation subtask within an online self-supervised loop. The core idea is to drive a multi-scale feature translator via weak supervision from the LR source to estimate cross-modal deformation fields and align the guide, while employing a content-aware reference filter that conducts discriminative self-enhancement purely on the source feature using the aligned guide as reference, thereby eliminating redundant modality-specific artifacts.

Method

Overall Architecture

RobSelf operates in an online self-supervised fashion on each individual test pair. Given an input pair consisting of an LR source \(\mathbf{I}_{source}^{LR} \in \mathbb{R}^{h \times w \times \phi}\) and an HR guide \(\mathbf{I}_{guide}^{HR} \in \mathbb{R}^{H \times W \times \psi}\), the source image is first bilinearly upsampled to match the spatial resolution of the guide. Shallow feature extraction layers yield source feature \(\mathbf{F}_{source}\) and guide feature \(\mathbf{F}_{guide}\). The architecture then coordinates two primary components: the Misalignment-Aware Feature Translator estimates a multi-scale cross-modal deformation field to output an aligned guide feature \(\mathbf{F}_{guide}^{Aligned}\) along with an auxiliary translated prediction \(\mathbf{I}_{pred}^{Trans}\); concurrently, the Content-Aware Reference Filter uses \(\mathbf{F}_{guide}^{Aligned}\) as reference to discriminatively filter and self-enhance \(\mathbf{F}_{source}\), producing an enhanced feature \(\mathbf{F}_{source}^{Enhanced}\) from which the final SR prediction \(\mathbf{I}_{pred}^{SR}\) is derived. The entire network is supervised solely by downsampling consistency against the LR source.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input: LR Source + HR Guide"] --> Feat["Multi-Modal Feature Extraction"]
    Feat --> Trans["Misalignment-Aware Feature Translator<br/>Deformation Estimation & Weakly-Supervised Translation"]
    Trans --> Align["Aligned Guide Feature F_guide_aligned"]
    Align --> Filt["Content-Aware Reference Filter<br/>Gradient Importance & Reference-Based Self-Enhancement"]
    Feat -->|Source feature stream| Filt
    Filt --> Head["SR Prediction Head generating I_pred_sr"]
    Trans --> TransHead["Translation Head generating I_pred_trans"]
    Head --> Loss["Downsampling Consistency Loss<br/>Supervised solely by LR Source"]
    TransHead --> Loss

Key Designs

1. Weakly-Supervised Misalignment-Aware Feature Translator: Multi-Scale Deformation and Modality Translation

To resolve cross-resolution and cross-modal alignment without ground-truth deformation fields, the translator models spatial misalignment end-to-end within the SR objective. It adopts an \(i\)-level (\(i=3\sim 5\)) encoder-decoder architecture that ingests the concatenated source and guide features. Through progressive \(2\times\) downsampling and upsampling, it captures deformation from localized offsets up to large-scale disparities, outputting an \(H \times W \times N\) deformation field \(\mathbf{D}_{G2S}\). Based on \(\mathbf{D}_{G2S}\), a spatial alignment layer warps \(\mathbf{F}_{guide}\) into \(\mathbf{F}_{guide}^{Aligned}\). Two architectural variants are presented: RobSelf-De, which employs deformable convolution, and RobSelf-Re, which uses direct spatial resampling: $$ \mathbf{F}{guide}^{Aligned}(p) = \begin{cases} \sum}^{k^2} w_i \mathbf{F{guide}(p + p_i + \Delta p_i), & \text{RobSelf-De} \ \mathbf{F} $$ where }(p + \Delta p), & \text{RobSelf-Re} \end{cases\(p\) denotes pixel coordinates, \(p_i\) and \(w_i\) are regular sampling positions and learnable weights in a \(k\times k\) kernel, and offset vectors are derived from \(\mathbf{D}_{G2S}\). The aligned feature is projected via a prediction head to generate an auxiliary prediction \(\mathbf{I}_{pred}^{Trans}\), supervised to mimic the source modality. This translation objective provides the vital inductive bias: driving the guide to reconstruct the source compels the deformation field to align shared geometries while naturally borrowing context to synthesize structures missing in the guide due to field-of-view differences or occlusions.

2. Content-Aware Reference Filter: Gradient-Guided Discriminative Self-Enhancement

Even after geometric alignment, the guide feature retains modality-specific redundant information (e.g., surface texture or illumination variations in RGB that do not correspond to geometric depth discontinuities). Direct concatenation or standard joint filtering inevitably causes texture-copying artifacts. To resolve this, the filter performs reference-based self-enhancement exclusively using source pixels, treating the aligned guide purely as a reference for computing attention weights. The filter first calculates an importance map \(\mathbf{M}_{imp} \in \mathbb{R}^{H \times W}\) via channel-averaged spatial gradients: $$ \mathbf{M}{imp} = \frac{1}{C}\sum}^{C} |\nabla(\mathbf{F{source}^c)| $$ An adaptive threshold \(\tau = \eta \cdot \mathbb{E}[\mathbf{M}_{imp}]\) segments the feature map. For high-gradient pixels (\(\mathbf{M}_{imp}(i) > \tau\)) representing salient structural edges, a larger kernel \(w_i^{imp} \in \mathbb{R}^{m \times m}\) aggregates broader contextual support. For low-gradient smooth areas (\(\mathbf{M}_{imp}(j) \le \tau\)), a smaller kernel \(w_j^{l\text{-}imp} \in \mathbb{R}^{n \times n}\) applies lightweight smoothing. Normalized correlation weights between neighboring source features \(\mathbf{F}_{source}(l)\) and the reference feature \(\mathbf{F}_{guide}^{Aligned}(i)\) are computed as: $$ w}^{imp} = \frac{\exp(\mathbf{F{source}(l)^\top \mathbf{F}}^{Aligned}(i))}{\sum_{l' \in \mathcal{Ni} \exp(\mathbf{F} $$ The enhanced feature }(l')^\top \mathbf{F}_{guide}^{Aligned}(i))\(\mathbf{F}_{source}^{Enhanced}(i) = \sum_{l \in \mathcal{N}_i} w_{i,l}^{imp} \mathbf{F}_{source}(l)\) updates each source position purely from source content guided by high-frequency reference correlations, effectively guarding the source modality against redundant guide artifacts.

Loss & Training

RobSelf is optimized online per image pair for 1000 iterations using the Adam optimizer with an initial learning rate of \(10^{-3}\), decayed by a factor of 0.9998 every 5 iterations. The overall loss combines SR reconstruction consistency and translation consistency against the LR source: $$ \mathcal{L} = \mathcal{L}{sr} + \lambda \mathcal{L}} = | f_{down}(\mathbf{I{pred}^{SR}) - \mathbf{I} |}^{LR1 + \lambda | f}(\mathbf{I{pred}^{Trans}) - \mathbf{I} |_1 $$ where }^{LR\(f_{down}\) is average pooling matching the SR scaling factor, and \(\lambda\) is set to 1. The optimization requires neither training sets nor ground-truth depth/NIR labels.

Key Experimental Results

Main Results

The authors benchmarked RobSelf on three distinct scenarios: synthesized misaligned RGB-guided depth SR (Middlebury), real-world misaligned RGB-guided depth SR (Azure Kinect DK dataset featuring physical displacements and viewpoint variations), and real-world misaligned RGB-guided NIR SR (incorporating object motion). Baseline methods lacking built-in alignment were evaluated with and without pre-alignment using the state-of-the-art MINIMA registration model.

Table 1: Synthetic and Real-World Misaligned RGB-Guided Depth SR (from original Table 1 & Table 2)

Dataset / Task Metric RobSelf-De (Ours) RobSelf-Re (Ours) Best Self-Supervised Baseline (w/ Pre-align) Best Supervised Baseline (w/ Pre-align) Improvement (vs Best Self-Sup)
Synthetic Depth \(\times 4\) RMSE \(\downarrow\) 1.43 1.52 1.88 (MMSR) 2.06 (C2PD) -0.45 (-23.9%)
Synthetic Depth \(\times 8\) RMSE \(\downarrow\) 2.49 2.57 2.79 (MMSR) 3.15 (C2PD) -0.30 (-10.8%)
Real-World Depth \(\times 2\) RMSE \(\downarrow\) 2.18 2.23 2.53 (CMSR, w/o pre-align) / 2.74 (SSGNet) 2.98 (SGNet) -0.35 (-13.8%)
Real-World Depth \(\times 4\) DSS \(\uparrow\) 8.6 8.4 6.7 (CMSR, w/o pre-align) / 5.6 (MMSR) 5.1 (SGNet) +1.9 (+28.4%)

Table 2: Real-World Misaligned RGB-Guided NIR SR (from original Table 3)

Dataset / Task Metric RobSelf-De (Ours) RobSelf-Re (Ours) CMSR (Self-Sup) MMSR* (Self-Sup) Fusion-CAP (Supervised) Margin
Real-World NIR \(\times 2\) RMSE \(\downarrow\) 3.12 3.09 3.42 8.91 4.40 -0.33 RMSE vs CMSR
Real-World NIR \(\times 4\) NIQE \(\downarrow\) 7.60 7.63 8.05 10.50 8.95 -0.45 NIQE (improved perceptual quality)

Ablation Study

On the real-world misaligned RGB-depth dataset (\(\times 2\) SR, Task II), the isolated and combined contributions of the translator (T) and filter (F) were ablated using RobSelf-De.

Table 3: Ablation Analysis of Translator and Filter Components (from original Table 4)

Configuration Translator (T) Filter (F) RMSE \(\downarrow\) Mechanism & Analysis
1. Baseline \(\times\) \(\times\) 2.55 Direct \(1\times 1\) conv fusion of unaligned features; severe structural blurring
2. w/ only T \(\checkmark\) \(\times\) 2.37 Features aligned by translator but fused naively; redundant texture leaks
3. w/ only F \(\times\) \(\checkmark\) 2.31 Filter operates without aligned guidance; false reference structures injected
4. Full (Ours) \(\checkmark\) \(\checkmark\) 2.18 Joint translator alignment + reference-based self-enhancement achieves SOTA

Key Findings

  • Compounding gains from joint translation and filtering: The ablation demonstrates that adding only the translator drops RMSE from 2.55 to 2.37, while adding only the filter reaches 2.31. Combining both yields a significant drop to 2.18, proving that accurate alignment and redundancy-free self-enhancement are mutually reinforcing.
  • Vulnerability of decoupled pre-alignment in wild settings: Two-stage pipelines fail drastically under complex real-world conditions (e.g., MMSR + MINIMA degrades to an RMSE of 8.91 on NIR \(\times 2\)). The massive cross-modal and cross-resolution gap impedes standalone registration, whereas RobSelf's joint optimization handles arbitrary misalignment gracefully.
  • Superior runtime efficiency: Efficiency evaluations (Table 5) indicate that on Task III (NIR \(\times 2\)), RobSelf-Re requires only 64 seconds per pair on an NVIDIA A100 GPU, operating 15.3\(\times\) faster than P2P (982s) and over \(2.5\times\) faster than MMSR (164s) and SSGNet (181s), all while maintaining a compact footprint of 0.65Mโ€“1.07M parameters.

Highlights & Insights

  • Contextual synthesis of missing guide geometries: In scenarios where partial scene content is completely absent from the guide image due to occlusions or camera FoV discrepancies (e.g., the edge of a pot or cactus), the translator implicitly borrows contextually related features to reconstruct the missing structure in \(\mathbf{F}_{guide}^{Aligned}\).
  • Strictly insulated self-enhancement: Rather than mixing cross-modal features into a blended space, the filter forces all output pixels to be convex combinations of source pixels alone. This structural isolation guarantees that guide colors and illuminations will not bleed into the target modality.
  • Broad portability across sensing domains: The self-supervised online formulation bypasses domain-shift vulnerabilities, offering an out-of-the-box solution for guided hyperspectral sharpening, ToF depth completion, and multi-modal medical imaging without paired training sets.

Limitations & Future Work

  • Lack of instantaneous inference capability: Although requiring only ~64โ€“122 seconds per pair (up to 15.3\(\times\) faster than prior self-supervised baselines), online iterative optimization remains unsuitable for hard real-time tasks such as autonomous drone navigation or high-frequency SLAM.
  • Potential failure under extreme non-rigid transformations: When local disparities or rapid non-rigid deformations exceed the receptive field of the multi-scale encoder, the self-supervised downsampling consistency loss may encounter local minima.
  • Future directions: Integrating meta-learned priors or lightweight test-time adaptation (TTA) strategies could accelerate convergence from 1000 iterations to under 20 iterations, narrowing the gap toward real-time deployment.
  • vs Two-Stage Registration + SR (e.g., MINIMA + MMSR): Independent registration lacks downstream restoration awareness and suffers from cross-resolution mismatches; RobSelf unifies alignment and super-resolution into an online end-to-end framework.
  • vs CMSR (CVPR 2021): CMSR uses off-the-shelf RGB spatial transformers on synthetic pseudo-pairs without modeling cross-modal dependencies, causing distorted boundaries; RobSelf's multi-scale translator handles complex real-world misalignment robustly.
  • vs MMSR (ECCV 2022) / SSGNet (AAAI 2023): Prior SOTA self-supervised models assume perfect pixel alignment and break down in the presence of sensor offsets; RobSelf establishes a new benchmark for unaligned cross-modal restoration.

Rating

  • Novelty: โญโญโญโญโ˜† Elegantly recasts unsupervised alignment as a weakly-supervised translation subtask combined with reference-based self-enhancement.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive validation spanning synthetic depth, real-world multi-sensor depth, and real-world NIR, paired with efficiency and feature visualizations.
  • Writing Quality: โญโญโญโญโญ Rigorous narrative flow, crisp problem formulation, and clean mathematical notations.
  • Value: โญโญโญโญโญ Highly impactful for real-world robotics, remote sensing, and multi-sensor systems operating in uncalibrated or shifting environments.