U2Flow: Uncertainty-Aware Unsupervised Optical Flow Estimation

Conference: CVPR 2026 arXiv: 2604.10056 Code: https://github.com/sunzunyi/U2FLOW Area: Video Understanding / Optical Flow Estimation Keywords: Optical Flow Estimation, Uncertainty Estimation, Unsupervised Learning, Recurrent Networks, Augmentation Consistency

TL;DR

U2Flow is the first recurrent unsupervised framework that jointly estimates optical flow and per-pixel uncertainty. Through augmentation-consistency-based decoupled uncertainty learning and uncertainty-guided bidirectional flow fusion, it achieves unsupervised state-of-the-art performance on KITTI and Sintel.

Background & Motivation

Background: Deep recurrent models based on all-pairs correlation (e.g., RAFT) achieve state-of-the-art results under full supervision, but acquiring large-scale accurate optical flow annotations is prohibitively costly, motivating unsupervised research.

Limitations of Prior Work: (1) Unsupervised models produce inaccurate estimates in occluded regions, textureless areas, and under large displacements—errors that are catastrophic for downstream tasks. (2) Uncertainty estimation in unsupervised settings is severely underdeveloped: direct supervision signals are absent, and it remains unclear how to effectively leverage uncertainty to improve flow estimation.

Key Challenge: A model must not only predict motion but also quantify its confidence in those predictions—yet without ground truth, how can a model be taught to assess its own reliability?

Goal: Achieve joint estimation of optical flow and uncertainty within a purely self-supervised framework, and use uncertainty feedback to improve flow estimation.

Key Insight: Exploit the inconsistency of model predictions under data augmentation as a self-supervised signal for uncertainty.

Core Idea: When a model produces inconsistent predictions under different perturbations, low-confidence regions are exposed—this inconsistency itself serves as a strong signal for uncertainty.

Method

Overall Architecture

The framework inherits the core design of RAFT (feature extraction → 4D correlation volume → recurrent update) and introduces an uncertainty estimation head and an uncertainty-aware refinement module. Training employs a photometric loss, a smoothness loss, and an augmentation-consistency-based uncertainty loss. At inference, uncertainty-guided bidirectional flow fusion is applied to improve robustness.

Key Designs

1. Decoupled Uncertainty Learning Strategy:

   • Function: Generate uncertainty supervision signals without ground truth.
   • Mechanism: A forward pass produces flow estimate \(\mathbf{F}_{1\to 2}\). Strong appearance/spatial augmentations are applied to the image pair to obtain \((\hat{I}_1, \hat{I}_2)\), from which a new flow estimate \(\hat{\mathbf{F}}'_{1\to 2}\) is computed. The discrepancy \(\hat{D}^{(k)} = \|\hat{\mathbf{F}} - \hat{\mathbf{F}}'^{(k)}\|_1\) serves as the uncertainty target. A Laplace likelihood MLE objective is used: \(\tilde{\ell}_{unc} = \sqrt{2}\exp(-\frac{1}{2}\alpha^{(k)})\hat{D}^{(k)} + \frac{1}{2}\alpha^{(k)}\), where \(\alpha = \log\sigma^2\). Critically, \(\hat{D}\) is detached from the computation graph to prevent gradient leakage.
   • Design Motivation: Unlike supervised methods that couple uncertainty and flow in a single MLE objective, the decoupled design prevents the uncertainty loss from interfering with flow estimation.

2. Uncertainty-Aware Refinement Module:

   • Function: Guide iterative flow refinement using predicted uncertainty.
   • Mechanism: Uncertainty weights \(\mathbf{s}^{(k)} = \phi(-\alpha^{(k)})\) are element-wise multiplied with flow features to produce scaled features \(\tilde{\mathbf{f}}^{(k)} = \mathbf{f}^{(k)} \odot \mathbf{s}^{(k)*}\). The original features, scaled features, and uncertainty map are then concatenated and passed through a convolutional head to output the flow residual.
   • Design Motivation: Features in high-uncertainty regions should be suppressed to reduce their negative influence on refinement.

3. Uncertainty-Guided Bidirectional Flow Fusion:

   • Function: Use the uncertainty of forward and backward flows to mutually correct each other.
   • Mechanism: Between the uncertainty maps of the forward and backward flows, the more reliable direction is selected for fusion, replacing the conventional occlusion-mask-based strategy. Uncertainty maps more accurately identify high-error regions.
   • Design Motivation: Traditional occlusion masks are binary and imprecise; continuous uncertainty values provide finer-grained reliability indicators.

Loss & Training

Total loss = photometric loss (census + SSIM + L1) + edge-aware smoothness loss + uncertainty-guided regional smoothness loss + augmentation-consistency uncertainty loss. On KITTI, an additional uncertainty-guided homography smoothness loss is applied.

Key Experimental Results

Main Results

Dataset	Metric	U2Flow	Prev. Unsupervised SOTA	Gain
KITTI 2015	Fl-all	SOTA	—	Significant
Sintel Clean	EPE	SOTA	—	Significant
Sintel Final	EPE	SOTA	—	Significant

Ablation Study

Configuration	Key Metric	Notes
w/o uncertainty estimation	Accuracy drops	Baseline RAFT
w/o decoupled design	Training unstable	Gradient leakage
w/o uncertainty refinement	Accuracy drops	Uncertainty not utilized
w/o bidirectional fusion	Worse in occluded regions	Traditional mask inferior to uncertainty
Full U2Flow	Best	All components synergize

Key Findings

• The decoupled design is critical for training stability—the detach operation prevents the uncertainty loss from interfering with the flow branch.
• Uncertainty maps more accurately identify high-error regions than traditional forward-backward consistency occlusion masks.
• Uncertainty-guided regional smoothness yields significant gains on KITTI (planar rigid motion scenarios).

Highlights & Insights

• "Model Self-Assessment" Paradigm: Without ground truth, augmentation consistency allows the model to expose its own uncertain regions—an elegant design choice.
• Importance of Decoupled Design: Explicitly separating uncertainty learning from flow regression avoids the instability of coupled objectives.
• Uncertainty as a Universal Signal: Uncertainty is used not only in the final output but also to dynamically modulate loss weighting and the refinement process during training.

Limitations & Future Work

• The augmentation consistency strategy assumes that augmentations are reasonable; extreme augmentations may introduce noisy supervision.
• The homography smoothness loss on KITTI relies on the planar rigidity assumption, limiting generalizability.
• The absolute calibration accuracy of the predicted uncertainty has not been validated (no ground-truth comparison available).

Related Work & Insights

• vs. ARFlow: ARFlow uses augmentation for knowledge distillation but does not estimate uncertainty; U2Flow repurposes augmentation consistency for uncertainty learning.
• vs. ProbFlow: ProbFlow employs variational inference for joint estimation but requires supervision; U2Flow achieves joint estimation in an unsupervised setting.

Rating

• Novelty: ⭐⭐⭐⭐ First unsupervised joint optical flow–uncertainty estimation
• Experimental Thoroughness: ⭐⭐⭐⭐ KITTI + Sintel dual benchmarks with detailed ablations
• Writing Quality: ⭐⭐⭐⭐ Method description is clear and well-organized
• Value: ⭐⭐⭐⭐ Uncertainty estimation holds significant importance for safety-critical applications

Related Papers

• [ICCV 2025] Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras
• [CVPR 2026] Enhancing Accuracy of Uncertainty Estimation in Appearance-based Gaze Tracking with Probabilistic Evaluation and Calibration
• [CVPR 2026] LAOF: Robust Latent Action Learning with Optical Flow Constraints
• [ICCV 2025] MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation
• [CVPR 2026] VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding