Frames2Residual: Spatiotemporal Decoupling for Self-Supervised Video Denoising¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/m1NGGi/F2R
Area: Image Restoration
Keywords: Video Denoising, Self-Supervised Learning, Spatiotemporal Decoupling, Residual Learning, Blind-Spot Network
TL;DR¶
To resolve the fundamental conflict between noise independence constraints and spatial texture recovery in self-supervised video denoising, F2R decouples learning into blind temporal consistency estimation and non-blind spatial texture refinement via a recorruption strategy, achieving state-of-the-art restoration on sRGB and raw videos.
Background & Motivation¶
Deep learning has brought significant breakthroughs to video denoising by utilizing temporal correlations across consecutive frames to identify coherent underlying structures and suppress stochastic noise fluctuations. However, the remarkable performance of existing video denoising models predominantly hinges on supervised training with paired clean and noisy video sequences. In crucial real-world scientific and industrial domains such as live-cell fluorescence microscopy, ultrafast transient imaging, or low-light cinematography, collecting paired noise-free ground truth sequences is prohibitively costly or physically impossible. Consequently, effective self-supervised video denoising trained directly on noisy footage has become an indispensable objective.
Existing self-supervised video denoising paradigms generally adapt image-based frameworks to the temporal domain, primarily falling into two schools of thought. Video Noise2Noise methods align neighboring frames via optical flow warping to construct self-supervision targets. However, spatial interpolation and grid resampling inevitably violate local pixel continuity and alter noise statistics, breaking the statistical independence assumption critical for self-supervision and producing severe ghosting artifacts and blur in dynamic scenes. Conversely, Video Blind-Spot Networks (Video BSNs) enforce noise independence by masking the central pixel or frame, estimating the clean signal strictly from surrounding contexts. While statistically sound, this complete exclusion of direct spatial evidence introduces pixel discontinuities, severs vital spatiotemporal correlations, and fatally restricts the network from restoring fine-grained spatial textures.
Enforcing strict blind-spot constraints for noise independence and utilizing direct spatial evidence for fine-grained texture restoration are fundamentally conflicting objectives in a single-stage network. This paper addresses this dilemma by breaking the single-stage coupled paradigm and explicitly decoupling temporal consistency modeling from spatial texture recovery across two consecutive training stages in the residual domain. Core idea: decouple self-supervised video denoising into a frame-wise blind temporal consistency estimation stage and a recorruption-driven non-blind spatial texture refinement stage, leveraging pre-trained image denoising priors as a structural baseline to safely reintroduce central spatial evidence for high-frequency residual recovery.
Method¶
Overall Architecture¶
The architecture of Frames2Residual (F2R) is established upon spatiotemporal decoupling and residual-domain optimization. The training pipeline is divided into two distinct sequential phases: Stage 1 trains a Blind Estimator (BE) using a frame-wise blind strategy, and Stage 2 trains a non-blind Spatial Refiner (SR) via an anchor-guided recorruption strategy. Both stages share an identical standard 4-level U-Net backbone operating on decomposed residuals, while their skip connections are tailored to their specific visibility regimes: Flow-Guided Attention Alignment Module (FAAM) in Stage 1 and Flow-Guided Deformable Alignment Module (FDAM) in Stage 2. Crucially, the dual-model decoupling is restricted entirely to training; during inference, the Blind Estimator is discarded and only the trained non-blind Spatial Refiner is deployed for direct, single-pass forward execution.
The end-to-end framework and data flow are outlined below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Noisy Video Y<br/>{y1, ..., yT}"] --> B["Preprocessing & Residual Decomposition<br/>D(yi) baseline & residual ri"]
B --> C["Joint Inputs & Flow Pyramid<br/>zi = cat(D(yi), ri) & V"]
C --> D["Stage 1: Blind Temporal Estimator (BE)<br/>Exclude center frame + FAAM alignment"]
D --> E["Temporally Consistent Anchor<br/>xs1 = D(yt) + BE temporal residual"]
E --> F["Stage 2: Center Recorruption Synthesis<br/>y't = xs1 + n' pseudo-noisy sequence Y'"]
F --> G["Stage 2: Non-blind Spatial Refiner (SR)<br/>Visible center frame + FDAM DCN alignment"]
G --> H["Inference Output<br/>x_final = SR(z, V) + D(yt)"]
Key Designs¶
1. Residual-Domain Formulation: Offloading Structural Modeling to Single-Image Denoising Priors
Directly predicting clean frames from noisy video sequences forces the network to expend substantial representational capacity on static low-frequency scene geometry rather than subtle cross-frame temporal textures. F2R addresses this by offloading static structure modeling to a pre-trained frozen image denoiser \(\mathcal{D}\) (e.g., NAFNet). For every noisy frame \(y_i\), the denoiser outputs a structural baseline \(\hat{x}_i = \mathcal{D}(y_i)\), and a high-frequency spatial residual is computed via subtraction: \(r_i = y_i - \hat{x}_i\). The network takes a concatenated joint representation \(z_i = \text{cat}(\hat{x}_i, r_i)\) for each time step. Because the deterministic image prior supplies clear structural references and suppresses noise ambiguity, the video network can dedicate its entire learning capacity exclusively to learning the inter-frame temporal residuals and high-frequency textures that were smoothed away by \(\mathcal{D}\).
2. Blind Temporal Estimating: Frame-wise Blind Masking with Attention Alignment
To strictly avoid noise contamination from the target frame without suffering from spatial pixel-wise checkerboard artifacts, Stage 1 introduces a Frame-wise Blind Strategy. The central frame \(y_t\) is entirely masked out from the input sequence, confining the observable temporal context strictly to adjacent frames \(\Omega_{\text{temporal}} = \{y_i\}_{i \neq t}\). A U-Net backbone processes each frame independently, deferring all cross-frame interaction to skip connections governed by the Flow-Guided Attention Alignment Module (FAAM). Using optical flow \(\{V_{i \to t}\}_{i \neq t}\) computed from the structural priors, FAAM warps neighboring feature representations toward the center frame. A channel attention mechanism dynamically reweights reliable frames to mitigate motion occlusion, while a subsequent spatial attention module filters out boundary warping distortions. The Blind Estimator \(f_\theta\) is supervised using the unobserved center noisy residual \(r_t = y_t - \mathcal{D}(y_t)\):
The resulting output is combined with the structural baseline to generate a temporally consistent anchor \(\hat{x}_{s1} = \hat{x}_t + f_\theta(\{z_i\}_{i \neq t}, V)\), which captures global motion continuity but lacks frame-specific spatial nuances due to center frame blindness.
3. Non-blind Spatial Refinement: Anchor Recorruption with Deformable Alignment
While the temporal anchor \(\hat{x}_{s1}\) secures cross-frame consistency, it misses the localized high-frequency textures removed by \(\mathcal{D}\). Stage 2 establishes a safe pathway to reintroduce central spatial evidence without compromising temporal stability via a Recorruption strategy. A pseudo-noisy sequence \(Y'\) is synthesized by replacing the clean anchor at the center position with a noise-injected version \(y'_t = \hat{x}_{s1} + n'\) (where \(n'\) matches the known or calibrated noise profile), while neighboring frames remain unchanged. Applying the frozen denoiser \(\mathcal{D}\) to \(y'_t\) produces a new baseline, allowing the missing high-frequency spatial residual to be precisely formulated as the deterministic gap: \(r'_t = \hat{x}_{s1} - \mathcal{D}(y'_t)\).
Crucially, the Spatial Refiner \(f_{\theta'}\) operates in a non-blind setting with full observation of the center input \(z'_t = \text{cat}(\mathcal{D}(y'_t), y'_t - \mathcal{D}(y'_t))\). Because the target frame is visible, the model can safely deploy the aggressive Flow-Guided Deformable Alignment Module (FDAM). Initialized with base optical flow offsets, FDAM concatenates center features \(F_t\), neighbor features, warped features, and flow vectors to regress fine-grained deformable convolutional offsets, capturing non-rigid sub-pixel motions. The Spatial Refiner is trained by minimizing:
Because \(\mathcal{D}\) behaves as an identical spatial filter across synthetic recorrupted samples and real inference data, the learned spatial residual mapping transfers directly. During inference, the Blind Estimator is discarded and the Spatial Refiner evaluates the original noisy input directly: \(\hat{x}_{\text{final}} = f_{\theta'}(\{z\}, V) + \mathcal{D}(y_t)\).
Loss & Training¶
For synthetic additive white Gaussian noise (AWGN, \(\sigma \in [10, 50]\)), models are trained on the DAVIS 2017 dataset with an \(L_1\) loss. For real-world raw video denoising (CRVD indoor dataset), models are trained with an \(L_2\) loss, and recorruption noise \(n'\) is drawn directly from calibrated sensor noise parameters matching each sequence's ISO level. Optimization uses the Adam optimizer with an initial learning rate of \(3 \times 10^{-4}\) decayed via a cosine annealing schedule over 200K iterations. The temporal window is set to \(T=9\) for synthetic experiments and \(T=7\) for raw videos. During inference, direct single-pass evaluation through the Spatial Refiner completely avoids cascaded latency.
Key Experimental Results¶
Main Results¶
F2R was comprehensively evaluated against traditional algorithms, supervised benchmarks, and competitive self-supervised methods on synthetic Gaussian noise datasets (DAVIS, Set8) and real raw video datasets (CRVD).
The quantitative comparisons (PSNR / SSIM) on DAVIS and Set8 benchmarks under various Gaussian noise levels are summarized below:
| Dataset | Method Type | Method | Average PSNR (dB) | Average SSIM | \(\sigma=30\) PSNR (dB) |
|---|---|---|---|---|---|
| DAVIS | Traditional | VBM4D | 32.39 | - | 31.65 |
| DAVIS | Supervised | NAFNet (single-frame) | 34.20 | 0.908 | 33.47 |
| DAVIS | Supervised | FastDVDNet | 34.64 | 0.919 | 34.04 |
| DAVIS | Supervised | FloRNN | 36.38 | 0.944 | 35.89 |
| DAVIS | Self-Supervised | UDVD | 34.76 | 0.920 | 34.09 |
| DAVIS | Self-Supervised | RDRF | 35.18 | 0.926 | 34.55 |
| DAVIS | Self-Supervised | TAP-T | 35.48 | 0.928 | 34.84 |
| DAVIS | Self-Supervised (Ours) | F2R (Ours) | 36.14 | 0.941 | 35.56 |
| Set8 | Traditional | VBM4D | 30.81 | - | 30.00 |
| Set8 | Supervised | FastDVDNet | 32.31 | 0.892 | 31.68 |
| Set8 | Supervised | FloRNN | 33.55 | 0.915 | 32.97 |
| Set8 | Self-Supervised | UDVD | 32.46 | 0.890 | 31.88 |
| Set8 | Self-Supervised | RDRF | 32.92 | 0.900 | 32.39 |
| Set8 | Self-Supervised | TAP-T | 33.95 | 0.902 | 33.42 |
| Set8 | Self-Supervised (Ours) | F2R (Ours) | 34.30 | 0.915 | 33.82 |
The table below lists quantitative PSNR (dB) performance across different ISO levels on the real-world CRVD indoor raw video dataset:
| Method Type | Method | ISO 1600 | ISO 3200 | ISO 6400 | ISO 12800 | ISO 25600 | Average PSNR (dB) |
|---|---|---|---|---|---|---|---|
| Supervised | NAFNet | 48.02 | 46.05 | 44.04 | 41.44 | 41.49 | 44.21 |
| Supervised | RViDeNet | 47.74 | 45.91 | 43.85 | 41.20 | 41.17 | 43.97 |
| Supervised | FloRNN | 48.81 | 47.05 | 45.09 | 42.63 | 42.19 | 45.15 |
| Self-Supervised | UDVD | 48.02 | 46.44 | 44.74 | 42.21 | 42.13 | 44.71 |
| Self-Supervised | RDRF | 48.38 | 46.86 | 45.24 | 42.72 | 42.25 | 45.09 |
| Self-Supervised | TAP | 48.85 | 47.03 | 45.11 | 42.44 | 42.33 | 45.15 |
| Self-Supervised (Ours) | F2R (Ours) | 49.41 | 47.52 | 45.66 | 43.03 | 42.82 | 45.69 |
Ablation Study¶
1. Contribution of Individual Stages in F2R (DAVIS and Set8 at \(\sigma=30\), CRVD at ISO=25600)
| Configuration | Pre-denoiser D | Stage 1 (Blind) | Stage 2 (Non-blind) | DAVIS PSNR / SSIM | Set8 PSNR / SSIM | CRVD PSNR (dB) | Note |
|---|---|---|---|---|---|---|---|
| Baseline Prior | ✓ | ✗ | ✗ | 33.47 / 0.904 | 31.81 / 0.869 | 41.49 | Single-frame NAFNet pre-denoiser alone |
| Stage 1 only | ✓ | ✓ | ✗ | 34.36 / 0.930 | 32.70 / 0.903 | 41.88 | Temporal anchor without spatial fine refinement |
| Stage 2 only | ✓ | ✗ | ✓ | 33.47 / 0.904 | 31.81 / 0.869 | 41.49 | Collapses to identity mapping without Stage 1 anchor |
| Full F2R | ✓ | ✓ | ✓ | 35.56 / 0.940 | 33.82 / 0.913 | 42.82 | Synergy of both decoupled stages yields peak metrics |
2. Impact of Core Components in Stage 1 (DAVIS, \(\sigma=30\))
| Configuration | Flow Alignment (FAAM) | Joint Residual Input (\(z\)) | PSNR (dB) | SSIM | Analysis |
|---|---|---|---|---|---|
| Plain U-Net | ✗ | ✗ | 25.06 | 0.731 | Direct frame stacking suffers severe misalignment |
| + FAAM | ✓ | ✗ | 28.22 | 0.870 | Flow-guided attention mitigates motion artifacts (+3.16 dB) |
| + Joint Inputs | ✗ | ✓ | 33.59 | 0.907 | Residual domain drastically simplifies optimization (+8.53 dB) |
| FAAM + Joint Inputs | ✓ | ✓ | 34.36 | 0.930 | Complementary designs enable high-quality temporal anchor |
3. Alignment Module Adaptation Across Stages (DAVIS, \(\sigma=30\))
| Stage 1 Module | Stage 2 Module | PSNR (dB) | SSIM | Architectural Insight |
|---|---|---|---|---|
| FAAM | FAAM | 35.37 | 0.939 | Uniform FAAM lacks sub-pixel deformation capacity in Stage 2 |
| FDAM | FDAM | 35.53 | 0.940 | Blind Stage 1 lacks center reference to stabilize DCN offsets |
| FAAM (Blind) | FDAM (Non-blind) | 35.56 | 0.940 | Stage-specific alignment achieves optimal performance |
Key Findings¶
- Sequential Stage Synergy is Indispensable: Training Stage 2 in isolation without Stage 1 produces 33.47 dB, identical to the raw NAFNet baseline. Without the stable temporal anchor extracted in Stage 1, recorruption training trivially degenerates into an identity mapping of the image denoiser. Conversely, cascading Stage 2 after Stage 1 provides a dramatic +1.20 dB boost on DAVIS, confirming the necessity of non-blind spatial refinement for restoring missing high-frequency details.
- Enormous Residual-Domain Dividend: Within Stage 1, replacing direct raw pixel sequences with joint inputs \(z_i\) that fuse image denoising baselines and high-frequency residuals yields an astounding 8.53 dB PSNR surge. Offloading low-frequency geometric learning frees all network capacity to model dynamic residual structures.
- Surpassing Supervised Baselines: On the Set8 benchmark, self-supervised F2R (34.30 dB) exceeds the fully supervised FloRNN model (33.55 dB) by 0.75 dB. On the CRVD raw benchmark, F2R consistently outperforms both supervised competitors (RViDeNet, FloRNN) and unsupervised state of the art (TAP), demonstrating exceptional noise modeling capability in real sensor noise distributions.
Highlights & Insights¶
- Decoupled Training with Streamlined Inference: By decoupling temporal consistency extraction from spatial texture recovery during training, F2R eliminates blind-spot degradation while discarding the Stage 1 network at test time, preserving high efficiency during real-world inference.
- Ingenious Recorruption Formulation: By leveraging the deterministic smoothing property of frozen image denoisers, injecting calibrated noise into the temporal anchor converts an otherwise ill-posed spatial texture recovery problem into a well-posed residual regression task.
- Visibility-Aware Alignment Scheme: The framework couples conservative attention filtering (FAAM) with the blind stage and precise deformable convolution (FDAM) with the non-blind stage, creating perfect harmony between model capacity and visual evidence.
Limitations & Future Work¶
- Prior Reliance on Image Denoiser Quality: F2R relies on a pre-trained frozen single-image denoiser to construct structural baselines. If the initial denoiser suffers from structural hallucinations or extreme oversmoothing under out-of-distribution noise, these structural errors can propagate into the residual domain.
- Sensitivity to Large Optical Flow Displacements: While FDAM accommodates fine-grained sub-pixel deformations, severe motion blur, occlusions, or extreme fast motion may degrade initial PWC-Net flow estimations, affecting feature aggregation quality.
- End-to-End Joint Fine-tuning Potential: Currently, the two stages are optimized sequentially. Future extensions could investigate adaptive gradient-routing mechanisms to facilitate end-to-end joint adaptation without collapsing the temporal anchor.
Related Work & Insights¶
- vs Video Noise2Noise (e.g., MF2F, RFR): Early methods warp neighboring frames to generate supervision targets, introducing interpolation artifacts that violate noise independence; F2R Stage 1 completely masks the center frame to strictly maintain statistical independence.
- vs Video BSN (e.g., UDVD, RDRF): Standard blind-spot networks strictly exclude the target pixel/frame during inference, resulting in smoothed textures and blurred lettering; F2R deploys a non-blind refiner during inference, fully exploiting central spatial evidence.
- vs TAP: TAP incorporates image denoising priors but trains in a coupled manner, struggling to reconstruct fine high-frequency patterns on dynamic textures; F2R uses a two-stage recorruption strategy that restores sharp textures with substantial quantitative gains.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decoupling video denoising into blind temporal consistency estimation and non-blind spatial refinement is conceptually elegant and highly effective.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across synthetic Gaussian and real-world raw datasets, complemented by detailed stage, module, and window-size ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, sound theoretical justifications, and clear visual and mathematical formulations.
- Value: ⭐⭐⭐⭐⭐ Closes and surpasses the gap with supervised video denoising models, providing a standout paradigm for self-supervised video restoration.