Skip to content

title: >- [Paper Note] Pol-CACTI: A System and dataset forHigh-Speed Polarized Video Compressive Imaging description: >- [ECCV 2026][Signal & Communication][Polarized Video SCI] Hardware-software co-designed Pol-CACTI integrates a DoFP polarization sensor with CACTI and PA-LBP reconstruction to achieve high-speed polarized video from a single snapshot. tags: - ECCV 2026 - Signal & Communication - Polarization Imaging System - Polarized Video Dataset - Snapshot Compressive Imaging date: 2026-09-19 content_hash: 8e3db8206450bbc3

Pol-CACTI: A System and dataset forHigh-Speed Polarized Video Compressive Imaging

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/kaeoqxhailesxya/Pol-CACTI
Area: Signal & Communication
Keywords: Polarization Imaging System, Polarized Video Dataset, Snapshot Compressive Imaging, Polarization Demosaicing, Learnable Back-Projection

TL;DR

To overcome the frame-rate bottleneck of conventional DoFP sensors and prevent severe error accumulation in decoupled two-stage pipelines, Pol-CACTI introduces a hardware-software co-designed polarization video SCI prototype, a curated 8,889-frame dynamic polarized video dataset, and a physics-driven joint reconstruction network featuring a Polarization-Aware Learnable Back-Projection (PA-LBP) module to achieve high-fidelity high-speed polarized video from a single compressive snapshot.

Background & Motivation

Polarization imaging captures essential physical surface properties beyond conventional radiant intensity, such as the Angle of Polarization (AoP) and Degree of Polarization (DoP). These cues play a vital role in 3D surface normal reconstruction, mechanical stress analysis, and material segmentation under low radiometric contrast conditions. However, acquiring high-speed polarization video remains fundamentally challenging due to physical sensing limitations. Division-of-focal-plane (DoFP) polarization cameras can capture single-shot mosaics, but their frame rates are severely restricted by sensor readout bandwidth and exposure time, leading to severe motion blur and loss of temporal dynamics in fast-moving scenes. Meanwhile, alternatives based on rotating waveplates or event cameras suffer from mechanical vibration, surface contamination, stray specular artifacts, and bandwidth drops during high-frequency event bursts.

Video snapshot compressive imaging (SCI, such as CACTI) offers a promising avenue by trading computation for transmission bandwidth via spatiotemporal mask modulation and temporal exposure integration. Nonetheless, naively marrying SCI with DoFP polarization sensing induces an intensely ill-posed inverse problem characterized by tightly entangled temporal multiplexing and spatial micro-polarizer mosaicing. In a decoupled two-stage pipeline—where SCI reconstruction is executed first, followed by standard polarization demosaicing—temporal inversion artifacts from the first stage are propagated and cross-talked across polarization channels during spatial interpolation. Crucially, polarimetric quantities like AoP and DoP are nonlinear functions of Stokes parameters, which drastically amplify minor reconstruction inaccuracies into physically inconsistent estimates.

To resolve the core challenge of spatiotemporal aliasing and error cascading, this paper presents a hardware-software co-designed framework. Core idea: physically align spatial DMD encoding units with DoFP macro-pixels, establish the first coupled spatiotemporal polarimetric forward model alongside a clean dynamic benchmark, and devise a physics-driven joint reconstruction network anchored by a Polarization-Aware Learnable Back-Projection (PA-LBP) module with physics-constrained loss functions to prevent error propagation across inversion and demosaicing.

Method

Overall Architecture

The optical hardware of Pol-CACTI utilizes a digital micromirror device (DMD) to modulate incoming dynamic light scenes with pseudorandom binary masks, relaying the coded snapshot onto a monochrome DoFP camera (Sony IMX250MZR) to record a single compressive mosaic measurement \(Y \in \mathbb{R}^{n_x \times n_y}\). The software reconstruction network operates under a physics-driven coarse-to-fine paradigm: an EfficientSCI video backbone first reconstructs the raw DoFP mosaic video sequence from the compressive snapshot; subsequently, deep feature refinement extracts high-frequency polarimetric details and passes them into the Polarization-Aware Learnable Back-Projection (PA-LBP) module to enforce measurement consistency, directly outputting full-resolution four-angle polarization video frames (\(I_0, I_{45}, I_{90}, I_{135}\)) and their derived Stokes parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single Polarized Compressive Measurement & Masks<br/>Y and M_t (H × W)"] --> B["Stage 1: EfficientSCI Backbone<br/>Coarse DoFP Mosaic Video Inversion"]
    B --> C["Stage 2: Deep Feature Refinement<br/>K=5 Residual Blocks for High-Frequency Polarization"]
    C --> D["Stage 3: PA-LBP Physical Correction<br/>Reprojection & Error Back-Projection"]
    D --> E["Stage 4: Physics-Constrained Loss Optimization<br/>Charbonnier & DoP-Weighted Cosine Loss"]
    E --> F["Output: Full-Resolution Dynamic Polarized Video<br/>I_0, I_45, I_90, I_135 and Stokes / AoP / DoP"]

Key Designs

1. 4×4 DMD Macro-Pixel Hardware Alignment: Eliminating Spatial Modulation Discrepancies

In optical compressive encoding, any sub-pixel misalignment between spatial mask patterns and sensor pixels causes angle-dependent spatial variation in modulation functions, which corrupts the linear energy conservation required for temporal recovery. The Pol-CACTI setup uses a precision multi-axis linear translation stage to physically align each \(4 \times 4\) DMD micromirror block with a corresponding \(4 \times 4\) sensor pixel cluster. This cluster spans exactly four complete \(2 \times 2\) DoFP macro-pixels (covering orientations \(0^\circ, 45^\circ, 90^\circ, 135^\circ\)), guaranteeing that the same temporal mask pattern \(M_t\) modulates all polarization channels uniformly without requiring post-hoc software geometric warping that could degrade high-frequency details.

2. Deep Feature Refinement: Multiscale Residual Enhancement for Fine Polarization Textures

Decoupled pipelines fail primarily because spatial demosaicing filters cannot distinguish between compressive temporal artifacts and genuine scene edges. The refinement network takes feature representation \(F_{sub}\) from the coarse SCI backbone, projects it via an initial convolutional layer to \(F_{head}\), and routes it through \(K=5\) standard residual blocks structured as \(x + \text{Conv}(\text{LeakyReLU}(\text{Conv}(x)))\). Followed by bilinear upsampling and convolutional filtering, this module explicitly learns high-frequency residual compensation \(I_{res}\) to restore structural details that were degraded by spatial downsampling and micro-polarizer mosaic patterns.

3. Polarization-Aware Learnable Back-Projection (PA-LBP): Anchoring Physical Consistency via Error Feedback

Standard feed-forward neural networks mapping features directly to demosaiced imagery are prone to hallucinations that diverge from physical observation constraints. The PA-LBP module embeds iterative back-projection error correction into deep feature learning: the network first sums the bilinearly upsampled polarization component tensor with the learned residual to obtain an initial estimate \(I_{pred}\); then, a learnable strided convolution \(\text{Conv}_{\downarrow}\) models the forward decimation to project \(I_{pred}\) back to the polarization component space, generating \(I_{proj}\); the discrepancy \(E_{diff}\) with respect to the input component tensor \(I_{pc}\) is calculated and back-projected via bilinear upsampling and convolution into an update map \(I_{corr}\):

\[I_{final} = I_{pred} + \text{Conv}(\text{Up}_{bi}(I_{pc} - \text{Conv}_{\downarrow}(I_{pred})))\]

This closed-loop feedback enforces measurement consistency with the raw DoFP sensor observations, anchoring the reconstruction to true physical states and dampening artifact propagation.

4. Physics-Constrained Loss Functions and Progressive Three-Stage Training: Resolving Phase Ambiguity

Directly applying \(\ell_1\) or \(\ell_2\) losses to the Angle of Polarization (AoP) fails due to \(\pi\)-periodicity phase wrapping and singularity issues in unpolarized regions. The objective combines Charbonnier loss terms across raw analyzer channels, Stokes total intensity \(S_0\), and DoP, paired with a DoP-weighted cosine difference loss designed for AoP:

\[\mathcal{L}_{AoP} = \mathbb{E}\left[\text{DoP}_{gt} \cdot (1 - \cos(2(\hat{\theta} - \theta_{gt})))\right]\]

This formulation naturally handles orientation modulo \(\pi\) and suppresses optimization gradients in regions where \(\text{DoP} \approx 0\) to avoid fitting noise. The network is trained through a three-stage progressive schedule: Stage 1 trains the SCI backbone for 300 epochs, Stage 2 freezes the backbone and trains the demosaicing branch for 100 epochs, and Stage 3 unfreezes all parameters for global fine-tuning over 20 epochs, ensuring smooth convergence across competing loss terms.

Key Experimental Results

Main Results

On the curated simulated test benchmark (\(512 \times 512 \times 48 \times 10\)) across compression ratios \(Cr \in \{8, 16, 32, 48\}\), Pol-CACTI demonstrates substantial gains in both intensity fidelity and polarimetric precision over two-stage baselines:

Compression Ratio \(Cr\) Method \(S_0\) PSNR (dB) \(S_0\) SSIM DoP PSNR (dB) DoP SSIM AoP MAE (\(^\circ\))
\(Cr = 8\) SCI Interp. 36.361 0.962 31.103 0.853 7.804
\(Cr = 8\) Polanalyser 36.384 0.964 25.382 0.684 16.955
\(Cr = 8\) PIDSR 35.886 0.964 27.460 0.737 14.311
\(Cr = 8\) PUGDiff 35.976 0.965 26.261 0.709 17.023
\(Cr = 8\) SwinCPD 35.923 0.961 29.063 0.807 9.537
\(Cr = 8\) Ours 42.122 0.987 31.327 0.874 7.279
\(Cr = 16\) SCI Interp. 34.228 0.942 28.651 0.800 9.805
\(Cr = 16\) Polanalyser 34.526 0.945 25.711 0.691 16.554
\(Cr = 16\) SwinCPD 34.146 0.942 28.396 0.796 10.245
\(Cr = 16\) Ours 37.190 0.963 29.587 0.840 8.164
\(Cr = 32\) SCI Interp. 32.807 0.925 27.628 0.780 11.033
\(Cr = 32\) SwinCPD 32.711 0.925 27.305 0.778 11.320
\(Cr = 32\) Ours 34.607 0.942 28.422 0.814 9.466
\(Cr = 48\) SCI Interp. 32.200 0.919 27.242 0.769 11.436
\(Cr = 48\) SwinCPD 32.131 0.920 27.082 0.771 11.647
\(Cr = 48\) Ours 33.507 0.936 27.634 0.796 10.213

Ablation Study

Ablation experiments conducted on the Pol-CACTI benchmark demonstrate the individual contributions of the training curriculum and architectural designs:

Category Variant \(S_0\) PSNR (dB) DoP PSNR (dB) AoP MAE (\(^\circ\)) Note
Training Strategy Serial Cascade 15.78 16.43 32.94 Naive sequential pipeline fails due to severe error cascading
Training Strategy Joint Training 33.16 24.18 30.26 Training from scratch struggles with optimization convergence
Training Strategy Stage 2 (Frozen) 35.91 29.89 14.48 Frozen backbone provides stable but suboptimal representation
Training Strategy Stage 3 (Ours Full) 39.79 30.64 9.33 Progressive curriculum yields the highest performance
Component Analysis w/o PA-LBP 30.26 21.55 39.09 Discarding back-projection leads to massive polarimetric degradation
Component Analysis Heavy ResNet 32.37 29.51 11.81 Parameter scaling alone cannot compensate for physical consistency
Component Analysis w/o Phys. Loss 39.47 21.99 14.50 Omitting physics loss causes DoP PSNR to plunge by 8.65 dB
Component Analysis Ours (Full) 39.79 30.64 9.33 Full architecture with physical feedback and constraints

Key Findings

  • PA-LBP is Essential for Physical Fidelity: Removing PA-LBP causes \(S_0\) PSNR to drop by 9.53 dB and increases AoP error from \(9.33^\circ\) to \(39.09^\circ\). Even scaling the demosaicing backbone with a Heavy ResNet yields only 32.37 dB in \(S_0\), confirming that algorithmic performance originates from the physical projection mechanism rather than model parameter capacity.
  • Physics Losses Safeguard Derived Polarization Parameters: Without physics-constrained losses, the network achieves an \(S_0\) PSNR of 39.47 dB, but DoP PSNR plummets to 21.99 dB, showing that general intensity fidelity does not automatically translate to accurate polarimetric estimation.
  • Progressive Training Prevents Optimization Collapse: Training end-to-end from scratch yields suboptimal results (\(S_0\) 33.16 dB) due to conflicting gradient updates between temporal inversion and spatial demosaicing, whereas the three-stage progressive curriculum stabilizes parameter updates.

Highlights & Insights

  • Hardware-Level Geometric Registration: Precise alignment of the \(4 \times 4\) DMD modulation unit with four DoFP macro-pixels eliminates spatial interpolation artifacts and spatial cross-talk at the hardware level.
  • Spatiotemporal Non-Local Regularization for Supervision: By leveraging multi-frame non-local dictionary learning (NLCSR), the authors effectively clean sensor noise without blurring high-frequency polarization edges, producing the first large-scale dynamic benchmark with 8,889 frames.
  • Generalizable Feedback for Multidimensional SCI: The PA-LBP formulation provides a closed-loop blueprint for integrating spatial mosaic decimation back-projection into deep feature architectures, offering a principled framework for color and hyperspectral compressive imaging.

Limitations & Future Work

  • Hardware Exposure Trade-offs: In the experimental prototype, the camera captures at 10 fps (effective 80 fps at \(Cr=8\)). Under ultra-low-light dynamic conditions, high DMD modulation rates may reduce photon counts and increase shot noise.
  • Static Coding Masks: The optical setup relies on pre-calibrated pseudorandom binary masks; exploring adaptive content-aware coding via Deep Optics could further improve compressive sampling efficiency.
  • Spectral-Polarization Coupling: The current implementation operates with a monochrome polarization sensor, leaving multi-spectral and polarization joint compressive video acquisition as an open research avenue.
  • vs. Decoupled Pipelines (e.g., SCI Interp. / Polanalyser / PIDSR): Decoupled methods treat temporal reconstruction and polarization demosaicing independently, which causes temporal compression artifacts to propagate into spatial interpolation. Pol-CACTI integrates them in a unified network with physical feedback, surpassing the best baseline by 5.7 dB in \(S_0\) PSNR at \(Cr=8\).
  • vs. Static Polarization Demosaicing (e.g., SwinCPD, PUGDiff): SOTA demosaicing models assume clean static mosaics and exhibit severe ringing and flickering when applied to SCI reconstructions; Pol-CACTI's closed-loop back-projection and temporal regularization ensure continuous and stable Stokes estimations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First end-to-end system and forward formulation for dynamic polarized video compressive imaging with DoFP sensors.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 8,889-frame dynamic dataset, multi-ratio simulated benchmarks, thorough ablations, and real hardware prototype validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, clean optical diagrams, and well-structured experimental presentations.
  • Value: ⭐⭐⭐⭐⭐ Fills a critical void in high-speed polarization video sensing and provides public datasets and code for the community.