A Benchmark for Heterogeneous Stereo Deblurring with Physically- and Epipolar-constrained Cross Attention

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/shinhoju/PECA
Project: https://pknu-vlab.github.io/peca/
Area: 3D Vision / Image Restoration
Keywords: Heterogeneous stereo deblurring, HSD benchmark, epipolar constraints, physical disparity bounds, asymmetric feature fusion

TL;DR

The paper introduces HSD for asymmetric wide/ultra-wide smartphone deblurring and PECA for cross-view retrieval within a physically plausible, directional epipolar window, improving NAFNet from 32.16 to 32.92 dB on HSD without the cost of global matching.

Background & Motivation

A smartphone's two cameras are not identical copies of the same sensor. The primary wide camera usually has better light collection and optical stabilization, while the ultra-wide module often lacks equivalent stabilization. Synchronized capture can therefore preserve sharp edges in one view while motion blur removes texture from the other. Single-image deblurring must infer missing detail from the damaged observation alone, whereas conventional homogeneous stereo restoration generally treats the two views as comparable inputs requiring symmetric reconstruction. Neither formulation makes hardware-induced quality asymmetry the central problem.

A sharp reference cannot simply be pasted onto the blurry target. The camera baseline introduces disparity, occlusions leave some target content without a counterpart, and repeated patterns create ambiguous matches. Existing heterogeneous-camera methods often perform dense correspondence estimation or iterative alignment on unrectified images. However, native smartphone stereo pipelines already produce approximately epipolar-aligned outputs. Searching arbitrary two-dimensional locations after that processing wastes computation and exposes blur-weakened similarity features to many impossible candidates.

The contribution addresses both evaluation and modeling: synchronized real videos preserve device characteristics, temporal integration controls ultra-wide blur, and optical camera parameters further restrict correspondence search along an epipolar line. Core idea: treat the sharp wide view as a one-way information source, retrieve details only within a geometrically valid local disparity window, and preserve the ultra-wide stream's own restoration pathway.

Method

Overall Architecture

Each training sample contains a blurry ultra-wide image, a synchronized sharp wide reference, and a sharp ultra-wide target. Inference uses only the first two. Separate encoder branches extract target and reference features; PECA performs asymmetric cross-view retrieval; residual fusion adds the retrieved information to the target restoration stream, which produces a sharp ultra-wide output rather than restoring both cameras symmetrically.

The pipeline has three key designs: HSD construction, a physical epipolar window, and confidence-responsive residual fusion. Dataset construction runs only during training and evaluation preparation. The supervision branch in the diagram is not an inference input and does not reveal the sharp ultra-wide answer to the model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Capture["Synchronized smartphone<br/>stereo videos"] --> Dataset["HSD construction"]
    Dataset --> Inputs["Blurry ultra-wide +<br/>sharp wide reference"]
    Dataset --> Target["Sharp ultra-wide target"]
    Inputs --> Encoders["Dual encoders"]
    Encoders --> Window["Physical epipolar window"]
    Window --> Fusion["Confidence-responsive<br/>residual fusion"]
    Encoders -->|Target feature identity path| Fusion
    Fusion --> Output["Decode restored ultra-wide"]
    Target -.->|Training and evaluation only| Output

Key Designs

1. HSD construction: retain device heterogeneity while controlling paired blur supervision

The authors capture synchronized wide and ultra-wide videos using the stereoscopic modes of devices including the iPhone 15 Pro Max, iPhone 16 series, and iPhone 17. They directly use native rectification rather than adding external calibration. The device pipeline compensates for focal-length and principal-point changes induced by autofocus and optical image stabilization, allowing the restoration model to exploit approximately horizontal epipolar lines. Here, real capture refers to the source videos, device processing, and handheld trajectories; it does not mean every training blurry image was acquired in a single real long exposure.

Consecutive ultra-wide frames are averaged around a target timestamp. The central ultra-wide frame becomes the sharp target, and the synchronized wide frame becomes the reference; Figure 2 illustrates an 11-frame integration. Instead of relying on 240 fps stereo capture, the authors control camera motion to keep inter-frame displacement moderate and collect 384 scenes for diversity. Integration along actual camera trajectories gives different projected motions at different depths, producing spatially varying, depth-dependent blur. Nevertheless, this construction does not fundamentally eliminate the risk of discrete ghosting when low-frame-rate integration encounters fast motion.

All three image types are bicubically resized from \(1080\times1920\) to \(720\times1280\). To exclude easy examples, the authors compute PSNR between the integrated blurry image and the sharp ultra-wide frame and discard pairs above 33 dB. They split by sequence rather than by individual frame: 256 training sequences and 128 test sequences yield 2,200 training pairs and 1,100 test pairs. This prevents adjacent-frame leakage across the split, but the filtering also means HSD emphasizes substantial blur rather than the full distribution of everyday camera degradation.

2. Physical epipolar window: exclude impossible correspondences before scoring similarity

At a shared feature scale, target and reference tensors both have shape \(H\times W\times C\). Three pointwise convolutions generate queries \(Q\) from ultra-wide features and keys \(K\) and values \(V\) from wide features. Information therefore flows explicitly from the sharp reference to the degraded target, rather than being exchanged symmetrically. Rectification confines candidate matches to the same row. Under the paper's camera arrangement, disparity also has a fixed direction, so searching both directions along the scanline is unnecessary.

The search scale follows the pinhole stereo disparity relation. Let \(B\) be the camera baseline, \(f\) the focal length in pixels, and the effective minimum acquisition distance be the larger of the two cameras' minimum focus distances. With \(S\) accounting for input resizing and feature downsampling, the paper gives:

\[ d_{\mathrm{eff}}=\max(d_{\min}^{\mathrm W},d_{\min}^{\mathrm{UW}}),\qquad D_{\mathrm{phys}}=\left\lceil\frac{fB}{d_{\mathrm{eff}}S}\right\rceil, \qquad D_{\max}\leq D_{\mathrm{phys}}. \]

The reported input-level disparity bound is approximately 70 pixels for the authors' setup. The geometric ceiling \(D_{\mathrm{phys}}\) must be distinguished from the operational window \(D_{\max}\). Encoded features already summarize spatial neighborhoods, so the authors select a smaller operational window and use \(D_{\max}=5\) across all backbones in the main experiments. This is an empirical choice guided by physical priors, not a proof that five discrete offsets necessarily cover every object. Near-field or out-of-focus objects and different camera hardware can change its validity.

For a target location \((i,j)\), let \(\Omega_{i,j}\) denote same-row reference candidates in the permitted disparity direction and within the restricted window. The cached extraction of Equation (4) has damaged inequalities and endpoints, so this note does not reconstruct an exact open or closed interval. Equation (5) and its surrounding explanation support the following normalized presentation of cross-view aggregation:

\[ \alpha_{i,j,k}=\operatorname{Softmax}_{k:(i,k)\in\Omega_{i,j}} \left(\frac{\cos(Q_{i,j},K_{i,k})}{\tau}\right),\qquad F_{\mathrm{cross}}(i,j)=\sum_{(i,k)\in\Omega_{i,j}}\alpha_{i,j,k}V_{i,k}. \]

The cached typesetting of Equation (5) is also damaged; the expression above is normalized from its textual definition, and the original paper remains authoritative for exact notation. The temperature \(\tau\), set to 0.01 by default, controls how strongly attention concentrates on the most similar candidates. A wide window invites competition from repeated patterns and weakly discriminative blurred features. Restricting candidates therefore changes matching quality as well as computation. Omitting channel factors, global two-dimensional, full-row, and PECA attention have complexities \(O((HW)^2)\), \(O(HW^2)\), and \(O(HWD_{\max})\), respectively. These are matching costs, not whole-network costs.

3. Confidence-responsive residual fusion: supplement target features instead of replacing them

The aggregated reference features pass through a learned projection and are added to the ultra-wide features before reconstruction. The cached Equation (6) loses its addition sign and ending, but Section 4.4 explicitly describes residual summation, supporting this presentation:

\[ F_{\mathrm{fuse}}=F_{\mathrm{UW}}+\operatorname{Proj}(F_{\mathrm{cross}};W_{\mathrm{cross}}). \]

The identity path matters because an occluded region may have no corresponding content in the wide view. The model can still recover information from the target instead of being forced to use reference texture at every location. The authors interpret concentrated attention as confident retrieval and diffuse attention as less informative guidance. The method does not separately estimate dense disparity, predict an explicit occlusion mask, or introduce an independent scalar confidence gate.

The interpretation is not a mathematical guarantee. Softmax weights always sum to one, and spreading them across candidates does not necessarily reduce the norm of the weighted value vector. If candidate values are similar, averaging can still produce a strong response. Consequently, falling back to single-view restoration should be understood as empirical behavior of learned features combined with a residual pathway, not as an automatic occlusion-rejection property of the aggregation formula.

A Worked Example

Consider text whose strokes are blurred in the ultra-wide image but remain sharp in the synchronized wide reference. This is a mechanism illustration, not an additional experiment. After encoding, the blurry location supplies a query and the reference supplies keys and values. PECA examines only local same-row candidates in the physically permitted direction. With the default \(D_{\max}=5\) and \(\tau=0.01\), the most compatible reference strokes receive stronger weights, are projected, and are added to the target features. The decoder reconstructs text using the ultra-wide geometry and its own features rather than copying an entire wide-camera patch.

If a nearby object is visible only in the ultra-wide view, all reference candidates may be unreliable. The target identity path still provides restoration evidence, but flat attention alone does not establish that the reference contribution becomes zero. Demonstrating reliable rejection would require additional occlusion-stratified evaluation or an explicit rejection mechanism, not merely an average attention visualization.

Loss & Training

All three backbones train for 400K iterations with batch size 8, AdamW, and cosine annealing. The learning rate, weight decay, and \(\beta\) settings are \(10^{-4}\), \(10^{-3}\), and \((0.9,0.9)\) for XYDeblur; \(3\times10^{-4}\), \(10^{-4}\), and \((0.9,0.999)\) for Restormer; and \(5\times10^{-4}\), \(10^{-3}\), and \((0.9,0.9)\) for NAFNet. Augmentations include random cropping, gamma adjustment, and shot/read noise.

XYDeblur and Restormer use \(128\times128\) crops, while NAFNet uses \(256\times256\) crops. The deepest features nevertheless have a fixed spatial size of \(32\times32\) for every backbone. NAFNet uses three levels with encoder block counts \([1,1,28]\) and decoder counts \([1,1,1]\); Restormer removes the downsampling operation before its fourth encoder stage. Cross-backbone results thus reflect the paper's configurations, not necessarily the original standard architectures.

The cached main text does not specify the exact restoration loss, its weights, or multi-scale supervision details. It would be unjustified to insert an L1, Charbonnier, frequency-domain, or additional geometric loss based on common practice. The paired restoration protocol and optimization settings above are supported; complete reproduction still requires checking the released code or supplementary material.

Key Experimental Results

Main Results

The following selection from Table 1 reports HSD test performance. PSNR is in dB; higher PSNR and SSIM are better. MACs are measured at \(720\times1280\) input resolution, with parameters in M and MACs in G.

Backbone	Variant	PSNR	SSIM	Params M	MACs G
XYDeblur	Single-view baseline	30.76	0.9444	4.92	1056.5
XYDeblur	Stereo concatenation	31.96	0.9595	7.37	1373.5
XYDeblur	PECA	32.22	0.9620	7.42	1376.4
Restormer	Single-view baseline	30.94	0.9444	12.59	2139.8
Restormer	Single-view channel expansion	31.17	0.9471	22.17	3754.7
Restormer	Stereo concatenation	31.83	0.9581	19.94	2851.3
Restormer	PECA	32.47	0.9636	20.05	2857.8
NAFNet	Single-view baseline	32.16	0.9579	17.08	832.6
NAFNet	Stereo concatenation	32.24	0.9604	33.38	1567.4
NAFNet	PECA	32.92	0.9669	34.17	1589.6

PECA improves over the respective single-view baselines by 1.46, 1.53, and 0.76 dB, and over stereo concatenation by 0.26, 0.64, and 0.68 dB. The latter comparison better isolates how reference information is used; the former also includes the benefit of access to a reference and additional branch capacity. Restormer's channel-expanded single-view model remains 1.30 dB below PECA despite higher computation, showing that parameter growth alone does not explain the improvement.

Ablation Study

Table 2 compares search spaces in the configuration corresponding to XYDeblur, holding the projection layers, cosine similarity, and \(\tau=0.01\) fixed. Module MACs and total model MACs must be kept distinct.

Search design	Candidate domain	PSNR	SSIM	Module MACs G	Total MACs G
Global	Full two-dimensional image	30.72	0.9439	852.2	2225.7
Full-row	Entire epipolar scanline	30.75	0.9438	7.6	1381.0
PECA	Physically bounded local epipolar window	32.22	0.9620	2.9	1376.4

Moving from global to full-row search sharply reduces computation but adds only 0.03 dB. Restricting the physical window adds another 1.47 dB. Relative to global attention, PECA reduces module MACs to approximately \(1/294\), but total model MACs decrease only from 2225.7G to 1376.4G. This is not a 294-fold whole-network speedup, and MACs alone do not establish smartphone frame rates.

Key Findings

• More comprehensive search is not necessarily better restoration. When blur weakens query discrimination, excluding impossible candidates is more useful than requiring the model to reject arbitrary positions. Full-row attention even remains below the single-view XYDeblur baseline of 30.76 dB.
• In Figure 7, XYDeblur reaches 32.33 dB with \(\tau=0.01\) and \(D_{\max}=9\), versus 32.22 dB with the default window of 5. A larger window can yield a modest improvement, but the main experiments retain a common cross-backbone setting.
• Natural handheld blur examples receive qualitative evaluation only because sharp ground truth is unavailable. HSD PSNR should not be presented as a real-long-exposure test score. The cache refers to supplementary occlusion and attention analyses but does not include their numerical results, so none are reconstructed here.

Highlights & Insights

• Physical priors can prune attention candidates without directly predicting depth. Here they improve both complexity and correspondence discrimination, making them a more task-specific intervention than merely replacing a fusion operator.
• Heterogeneous stereo restoration benefits from distinguishing the information source from the restoration target. Drawing keys and values from the sharper view while preserving the degraded target's own pathway matches asymmetric smartphone degradation.
• The data protocol delegates geometric alignment to the native device pipeline and makes blur restoration the controlled variable. This supports focused evaluation, but conclusions remain conditional on the rectification quality and acquisition protocol.

Limitations & Future Work

• Author-acknowledged boundaries: HSD does not explicitly model rolling shutter, and natural handheld examples lack ground truth. Temporal integration is not equivalent to the complete real exposure, sensor, and ISP image-formation process.
• Conditions behind the physical bound: Minimum focus distance describes a sharp-imaging condition, not an absolute lower bound on object distance. Near-field defocused objects, different baselines, and resizing or crop changes can violate the operational assumptions. Cross-device and close-range evaluation would be informative.
• No guaranteed confidence rejection: The residual path preserves target information but does not explicitly reject an incorrect reference. Attention with a no-match option, occlusion-aware gating, and separate metrics for reference-invisible regions are possible extensions proposed by this note, not implemented features of the paper.
• Efficiency is not demonstrated deployment: NAFNet+PECA still costs 1589.6G MACs, and the cache provides no phone latency, memory, or energy measurements. Three backbones establish useful architectural coverage, not real-time operation on arbitrary devices or superiority over every heterogeneous restoration system.

Related Work & Insights

• Compared with XYDeblur, Restormer, and NAFNet: These backbones reconstruct images, whereas PECA introduces a geometrically constrained sharp reference. Table 1 is most informative as a set of within-backbone comparisons, not as evidence for an entirely new restoration backbone.
• Compared with PASSRnet, iPASSR, and NAFSSR: Epipolar attention already exists in stereo super-resolution. PECA further limits candidates through optical bounds and disparity direction for asymmetric deblurring; attention along an epipolar line itself is not new here.
• Compared with Deep Hybrid Camera Deblurring and Asymmetric Dual-Lens Video Deblurring: Complementary restoration with heterogeneous cameras has precedents. This paper emphasizes a controlled benchmark under native rectification and a lightweight retrieval module. The cache does not report common-protocol quantitative comparisons against those systems, so a claim of comprehensive superiority would be unsupported.

Rating

• Novelty: 4/5. The combination of physical disparity limits, directional epipolar retrieval, and an asymmetric deblurring benchmark is concrete, while attention and residual fusion have established foundations.
• Experimental Thoroughness: 3/5. Three backbones, capacity controls, and search-space ablations are useful; real-blur quantitative evaluation, device-held-out testing, and deployment measurements remain limited.
• Writing Quality: 4/5. The task and experimental reasoning are clear, but the causal explanation connecting diffuse attention to reduced fusion magnitude needs greater precision.
• Value: 4/5. HSD and the plug-in design are useful for studying smartphone multi-camera restoration and geometric priors. These scores are the note author's subjective assessment.