PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications

Conference: ECCV 2026
Paper: ECCV Official
Full Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-4389.txt
Area: Others
Keywords: Polarization Vision, Polarization Demosaicking, Shape from Polarization, De-reflection from Polarization, Meta-Learning Feature Alignment

TL;DR

PolarAPP is the first framework to jointly optimize polarization demosaicking and downstream polarimetric applications (such as surface normal estimation and de-reflection), breaking the traditional bottlenecks of low-quality regrouped datasets and task-agnostic demosaicking through bi-level meta-learned feature alignment and equivalent imaging transformation priors.

Background & Motivation

Polarimetric imaging captures rich physical properties beyond scalar light intensity—such as surface normals, refractive indices, and subsurface scattering—enabling critical downstream computer vision applications including shape from polarization (SfP) and reflection removal (DfP). However, standard commercial division-of-focal-plane (DoFP) color polarization filter array (CPFA) sensors (e.g., Sony IMX250MZR) interleave a Bayer pattern with a \(2 \times 2\) micro-polarizer array, meaning each pixel measures only a single polarization orientation (0°, 45°, 90°, or 135°). To supervise downstream deep networks, prevailing pipelines routinely rely on an oversimplified shortcut: they extract and regroup pixels of the same orientation into four half-resolution channels and apply standard Bayer interpolation independently. This naïve preprocessing completely bypasses joint polarization demosaicking, inevitably halving spatial resolution and introducing severe phase and angle-of-polarization (AoP) errors that propagate downstream into distorted geometry and inaccurate normal maps.

Meanwhile, modern deep learning-based polarization demosaickers are strictly trained as standalone low-level image restoration modules, optimizing pixel-level \(\ell_1\) fidelity or perceptual metrics like LPIPS. While these objectives produce visually pleasant images, they remain task-agnostic and neglect the sensitive physical constraints demanded by downstream tasks. When practitioners attempt a simple two-stage cascade by feeding demosaicked images into a task model, conflicting gradient objectives and the absence of latent semantic communication prevent the demosaicker from adapting to downstream utility, leaving downstream performance capped.

These persistent performance gaps stem from two interrelated flaws: performing polarimetric tasks without proper demosaicking, and isolating demosaicking from downstream task objectives. Core idea: PolarAPP bridges this fundamental gap by jointly optimizing demosaicking and downstream tasks end-to-end, leveraging bi-level meta-learning to establish a task-adaptive feature alignment metric, an equivalent imaging transformation (EIT) self-supervised prior to decouple reconstruction from dataset artifacts, and a task-refinement stage to unlock high-resolution geometric and radiometric details.

Method

Overall Architecture

The inference pipeline of PolarAPP takes a single CPFA mosaic raw frame \(R\) as input to produce downstream task predictions \(Y\). The raw mosaic is initially Bayer-interpolated to yield half-resolution color polarization representations, which are then passed into a Restormer-based polarization demosaicker \(\mathcal{D}\) to reconstruct full-resolution four-channel polarization images \(I = \{I_{0^\circ}, I_{45^\circ}, I_{90^\circ}, I_{135^\circ}\}\). From these four linear channels, the system computes physically interpretable Stokes parameters \(S_0, S_1, S_2\), degree of linear polarization (DoLP), and angle of polarization (AoP). These cues, concatenated with spatial coordinates, feed directly into a task-specific network \(\mathcal{T}\) (e.g., normal estimation for SfP or reflection removal for DfP) to infer the final output \(Y\).

During training, directly backpropagating combined task and reconstruction losses induces severe objective mismatch and risks latent collapse. To establish robust synergy, PolarAPP introduces meta-learned feature alignment, an equivalent imaging transformation prior, and a high-resolution task refinement phase:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single-frame CPFA Raw Mosaic Input R"] --> B["Equivariant Imaging Transformation Prior Demosaicker D"]
    B --> C["Physical Polarization Extraction<br/>Stokes Parameters, DoLP and AoP"]
    C --> D["Meta-Learning Feature Alignment Bi-level Optimization"]
    D --> E["Downstream Task Prediction Network T"]
    E --> F["Task-Network Refinement Stage"]
    F --> G["High-Precision Downstream Predictions<br/>SfP Normal Map / DfP De-reflected Image"]

Key Designs

1. Equivariant Imaging Transformation Prior Demosaicker: Self-supervised physical constraint decoupling pseudo-ground-truth artifacts

Downstream polarization datasets (e.g., SfPUEL, PolarRGB) are constructed from real DoFP captures via naïve regrouping and downsampling, inherently carrying demosaicking artifacts and sensor noise. Supervising \(\mathcal{D}\) solely on these degraded targets forces the network to memorize spurious reconstruction patterns. To resolve this, PolarAPP incorporates an Equivariant Imaging Transformation (EIT) self-supervised prior. Because the forward imaging operator \(\mathcal{A}\) (combining CPFA subsampling and Bayer filtering) is deterministic and physically known, an ideal reconstruction operator \(\mathcal{D}\) must be equivariant to geometric spatial transformations \(\mathcal{G}\): transforming the full-resolution reconstructed polarization image by \(T_g \in \mathcal{G}\) (including integer translation, 10° rotation, and horizontal flipping) before re-sampling through \(\mathcal{A}\) should yield a demosaicked result identical to applying \(T_g\) directly to the original reconstruction:

\[\mathcal{D}(\mathcal{A}(T_g(I^{rec}))) = T_g(I^{rec})\]

The corresponding self-supervised loss \(\mathcal{L}_{eit}\) enforces physical self-consistency across diverse transformed views without requiring external ground truth. The total demosaicking loss becomes \(\mathcal{L}_d + \mathcal{L}_{eit}\), where \(\mathcal{L}_d\) incorporates \(\ell_1\) and gradient constraints over Stokes components and DoLP, alongside wrapped angular \(\ell_1\) supervision on AoP.

2. Meta-Learning Feature Alignment Bi-level Optimization: Goal-directed latent alignment preventing representation collapse

Directly cascading \(\mathcal{D}\) and \(\mathcal{T}\) with an end-to-end composite loss \(\mathcal{L}_d + \mathcal{L}_t\) suffers from gradient interference and lacks latent communication. PolarAPP introduces two lightweight feature transformation modules \(\{FT_1, FT_2\}\) to project the hierarchical features \(F_d^j\) and \(F_t^j\) (\(j \in \{1, 2, 3\}\)) into a shared metric space, measuring similarity via the feature alignment loss:

\[\mathcal{L}_{fa} = 3 - \sum_{j=1}^{3} \frac{\langle FT_1(F_d^j), FT_2(F_t^j) \rangle}{\|FT_1(F_d^j)\|_2 \cdot \|FT_2(F_t^j)\|_2}\]

However, imposing an unconstrained fixed alignment loss easily triggers representation collapse (e.g., trivial constant mappings that minimize \(\mathcal{L}_{fa}\) without benefiting the primary tasks). PolarAPP formulates a bi-level meta-learning optimization strategy to address this: - Inner Update: On a meta-training split, tentative parameters \((D', T')\) are generated by taking a single gradient exploration step driven solely by \(\mathcal{L}_{fa}\) while original model weights remain untouched; - Outer Update: On a separate meta-test split, the tentative networks \((D', T')\) are evaluated on true demosaicking and task losses (\(\mathcal{L}_d, \mathcal{L}_t\)). The resulting second-order gradients optimize the feature transformation modules \(\{FT_1, FT_2\}\).

This bi-level formulation guarantees that the alignment metric is updated strictly when it produces concrete improvements on the ultimate reconstruction and task goals. Crucially, the transformation modules \(\{FT_1, FT_2\}\) are discarded at inference, adding zero computational or memory overhead.

3. Task-Network Refinement Stage: Bridging the resolution gap and unleashing full physical cues

During joint training, the imaging operator \(\mathcal{A}\) must remain active to synthesize paired inputs matching downsampled dataset targets, which restricts the demosaicker to compressed representations and hinders task network convergence. To fully exploit the reconstructed fidelity, PolarAPP introduces a dedicated refinement phase upon convergence of joint training. Here, the demosaicker \(\mathcal{D}\) is completely frozen, and the operator \(\mathcal{A}\) is removed. The frozen demosaicker now processes uncompressed, full-resolution raw mosaics, feeding clean, artifact-free polarization tensors directly into \(\mathcal{T}\).

In this stage, the task network is fine-tuned using task supervision \(\mathcal{L}_t\) and alignment guidance \(\mathcal{L}_{fa}\). For SfP, ground-truth surface normals are bilinearly upsampled solely for spatial scale alignment. For DfP, the frozen demosaicker synthesizes full-resolution reference guidance. This decoupled refinement harmonizes training with actual deployment conditions and enables the task network to fully absorb sharp high-frequency angular boundaries.

Loss & Training

The training pipeline executes an epoch-wise three-stage alternating schedule: 1. Meta-Learning Phase: Alternates between inner exploratory updates and outer second-order gradient updates for \(M=200\) iterations to update \(\{FT_1, FT_2\}\); 2. Joint Learning Phase: Freezes \(\{FT_1, FT_2\}\) and updates \(\mathcal{D}\) and \(\mathcal{T}\) jointly with \(\mathcal{L} = \mathcal{L}_d + \mathcal{L}_{eit} + \lambda_t \mathcal{L}_t + \lambda_{fa} \mathcal{L}_{fa}\), setting \(\lambda_t = 20\), \(\lambda_{fa} = 100\) for SfP, and \(\lambda_{fa} = 10\) for DfP; 3. Task Refinement Phase: Removes operator \(\mathcal{A}\), freezes \(\mathcal{D}\), and optimizes \(\mathcal{T}\) using \(\mathcal{L}_t + \lambda_{fa} \mathcal{L}_{fa}\). All models are trained using the Adam optimizer with a learning rate of \(5 \times 10^{-5}\) across 100 epochs on NVIDIA RTX 4090 GPUs.

Key Experimental Results

Main Results

The authors evaluate PolarAPP across polarization demosaicking (Qiu benchmark), Shape from Polarization (SfPUEL dataset), and De-reflection from Polarization (PolarRGB dataset).

Polarization Demosaicking Comparison (Qiu Test Set):

Method	\(S_0\) PSNR (dB) ↑	\(S_0\) SSIM ↑	DoLP PSNR (dB) ↑	DoLP SSIM ↑	AoP MAE (°) ↓
DCPM [21]	42.98	0.976	37.92	0.918	6.456
PIDSR [56]	43.07	0.973	37.86	0.925	5.587
PolarAPP (Ours)	43.12	0.982	38.10	0.928	5.444

Downstream Application Quantitative Comparison (SfP and DfP):

Task	Method / Pipeline	11.25° Acc (%) ↑	22.5° Acc (%) ↑	30° Acc (%) ↑	Mean Angle Err (°) ↓	Median Angle Err (°) ↓	RMSE Angle Err (°) ↓
SfP	SfPUEL (CPFA Direct Regroup)	70.11	91.15	95.02	10.58	7.59	15.30
	PIDSR+ (Two-stage Retrained)	78.12	90.73	95.55	6.78	4.88	11.45
	PolarAPP (Ours Joint)	92.37	97.45	98.48	3.57	2.71	8.64

Task	Method / Pipeline	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	MUSIQ ↑
DfP	PolarFree (CPFA Direct Regroup)	22.44	0.868	0.132	60.31
	PIDSR+ (Two-stage Retrained)	22.41	0.859	0.124	60.79
	PolarAPP (Ours Joint)	22.90	0.871	0.129	61.29

(Note: PIDSR+ denotes retraining the downstream task network on top of fixed outputs from the SOTA demosaicker PIDSR.)

Ablation Study

The systematic ablation analysis under the unified experimental setting is summarized below:

Configuration	DoLP PSNR ↑	DoLP SSIM ↑	AoP MAE (°) ↓	SfP 11.25° Acc (%) ↑	SfP MAE (°) ↓	SfP RMSE (°) ↓	DfP PSNR (dB) ↑	DfP SSIM ↑	Description
w/o \(\mathcal{D}\) in training	/	/	/	78.14	6.01	13.40	22.26	0.851	Training task model directly on CPFA-regrouped input
Task-agnostic \(\mathcal{D}\)	38.13	0.927	5.502	76.91	5.87	12.98	22.40	0.846	Pretraining \(\mathcal{D}\) standalone and freezing it during task training
w/o \(\mathcal{L}_{eit}\)	36.75	0.885	8.011	86.12	5.73	12.57	22.46	0.869	Removing equivariant imaging transformation prior
Naïve Joint	38.43	0.924	5.658	82.44	6.09	11.86	22.37	0.858	Naïve end-to-end training using \(\mathcal{L}_d+\mathcal{L}_t\) without meta alignment
w/o Refinement	38.05	0.920	5.789	89.01	4.68	10.13	22.59	0.869	Omitting second-stage high-resolution task refinement
PolarAPP (Full)	38.10	0.928	5.444	92.37	3.57	8.64	22.90	0.871	Full joint pipeline: Meta-alignment + EIT + Refinement

Key Findings

• Crucial Need for Task-Aware Joint Optimization: Removing \(\mathcal{D}\) drops the strict 11.25° normal accuracy from 92.37% down to 78.14%, with mean angular error jumping from 3.57° to 6.01°. Notably, using a high-quality task-agnostic demosaicker (Task-agnostic \(\mathcal{D}\)) yields an even worse 11.25° accuracy of 76.91%. This proves that fidelity-oriented demosaicking distorts physical cues critical for normal recovery, whereas PolarAPP effectively aligns low-level reconstruction with high-level task goals.
• EIT Prior Stabilizes Physical Angles: Excluding \(\mathcal{L}_{eit}\) causes AoP error to surge from 5.444° to 8.011° and lowers DoLP PSNR by 1.35 dB. The equivariant constraint successfully prevents the model from overfitting to downsampled pseudo-ground-truth artifacts.
• Meta-Learning Prevents Gradient Conflict: Naïve joint training without meta-learned alignment degrades SfP mean angular error to 6.09° and 11.25° accuracy to 82.44%. The bi-level meta-update ensures that feature alignment occurs only along gradient directions that improve end objectives.
• Hyperparameter Sensitivity: The alignment weight \(\lambda_{fa}\) reaches optimal performance at 100 for SfP (which demands strict global geometric consistency) and at 10 for DfP (which requires delicate photorealistic texture balance).

Highlights & Insights

• Bridging Low-Level Restoration and Physical Sensing: Unlike traditional pipelines that separate low-level demosaicking from high-level vision, PolarAPP demonstrates that downstream physical tasks offer powerful structural supervision for low-level recovery, while task-aware demosaicking is essential to prevent error propagation.
• Degeneracy-Free Feature Alignment via Meta-Learning: Implementing bi-level optimization (MAML formulation) to train intermediate feature alignment transforms prevents latent representation collapse and guarantees zero extra parameter or runtime overhead during inference.
• Physical Equivariance over Pseudo-Labels: Exploiting deterministic forward imaging operators to form self-supervised equivariance constraints provides an effective strategy for learning inverse imaging problems where uncompressed ground truth is physically unobtainable.

Limitations & Future Work

• Computational Overhead of Second-Order Gradients: The meta-learning outer loop requires computing second-order Hessian-vector products, increasing training time and GPU memory footprints compared to standard networks.
• Disparate Task-Specific Weighting: The optimal alignment weight \(\lambda_{fa}\) spans an order of magnitude across tasks (\(\lambda_{fa}=100\) for SfP vs. \(10\) for DfP), indicating that adaptive, automated multi-task balancing schemes are needed for broader application.
• Static Scene Equivariance Assumption: The EIT prior assumes rigid geometric transforms on static scenes, which may not capture spatio-temporal polarization motion blur in highly dynamic environments.

Related Work & Insights

• vs PIDSR [56]: PIDSR treats demosaicking and super-resolution as a purely low-level image restoration task. PolarAPP shows that even state-of-the-art restoration models produce subtle phase errors that degrade downstream normal estimation by over 14% accuracy, validating the superiority of joint task-aware training.
• vs SfPUEL [32] & PolarFree [50]: These downstream architectures rely on raw regrouped mosaic inputs. PolarAPP provides a drop-in physical front-end that cleans up polarization measurements without altering downstream network heads, establishing new SOTA benchmarks across both tasks.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering joint optimization framework for polarization demosaicking and downstream tasks with meta-learned alignment and EIT priors]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across Qiu demosaicking, SfPUEL normal estimation, and PolarRGB de-reflection benchmarks with deep ablation studies]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, comprehensive motivation, and intuitive structural diagrams]
• Value: ⭐⭐⭐⭐⭐ [Offers a foundational paradigm shift for DoFP computational imaging from isolated restoration to task-driven physical reconstruction]