FlowPainter: Inpainting Optical Flow via Confidence-Guided Completion¶

Conference: ECCV 2026
Paper: ECCV Official
Code: Not publicly available
Area: Image Generation
Keywords: Optical Flow, Diffusion Models, Confidence Guidance, Image Inpainting, Residual Guidance

TL;DR¶

FlowPainter reformulates dense optical flow generation as confidence-guided soft inpainting, where a lightweight confidence-aware network predicts reliable simple-flow priors to initialize diffusion states and inject time-decayed residual guidance, reducing denoising burden and accelerating training convergence by over \(3\times\).

Background & Motivation¶

Optical flow estimation is a foundational vision task aiming to infer dense pixel-wise motion vectors between adjacent frames, underpinning video understanding, autonomous driving, and low-level restoration. Existing deep learning paradigms predominantly fall into two categories: iterative optimization and diffusion-based generative modeling. Iterative approaches, exemplified by RAFT, recurrently update flow predictions based on all-pairs correlation volumes, achieving remarkable accuracy under normal motions. However, when faced with extreme displacements, severe occlusions, or complex non-rigid deformations, their correlation matching and recurrent updates easily degrade due to error propagation and the absence of explicit uncertainty modeling. Conversely, diffusion-based methods such as FlowDiffuser and DDVM model the multi-modal distribution of flow fields, demonstrating superior capability in synthesizing plausible motions across ambiguous and extreme regions.

Nevertheless, existing diffusion-based optical flow models invariably denoise the entire dense flow field from pure Gaussian noise. In physical scenes, motion difficulty is inherently heterogeneous: while foreground moving objects may exhibit intricate non-rigid deformations, background regions translating with camera motion or smooth objects typically constitute easy, small-displacement flow. These simple regions can be reliably and cheaply estimated by a lightweight discriminative network. Forcing a generative diffusion model to reconstruct simple flow from random noise across the whole image imposes an unnecessary denoising burden. This leads to severe training instability, slow convergence (often requiring weeks on multi-GPU clusters), and a lack of early-stage structural anchors.

Observing this complementary behavior between lightweight discriminative estimators on simple motions and diffusion models on challenging motions, this work breaks away from the conventional "full-field noise denoising" paradigm. Core idea: leverage a lightweight confidence-aware network to predict rough flow and explicitly supervised confidence masks as a reliable prior, and cast dense flow estimation into a confidence-weighted initialization and time-decayed residual inpainting process within a diffusion framework.

Method¶

Overall Architecture¶

FlowPainter consists of two primary modules: a lightweight Confidence-Aware Network and a Flow Inpainting Network. Given two consecutive input frames, the Confidence-Aware Network first estimates a rough flow field and a pixel-wise confidence mask with minimal computational overhead. Their element-wise product forms the simple-flow prior. Subsequently, the Flow Inpainting Network utilizes this prior to perform confidence-gated soft initialization at the initial diffusion step. Throughout the iterative reverse diffusion process, a Time Step Decay Module injects confidence-gated residual guidance, cooperating with standard image conditioning features to progressively refine challenging, occluded, and fine-detailed motion boundaries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Two Consecutive Frames Input<br/>Frame 1 & Frame 2"] --> B["Confidence-Aware Prediction<br/>Lightweight network outputs rough flow & confidence"]
    B --> C["Simple-Flow Prior Construction<br/>Pixel-wise gating extracts reliable regions"]
    C --> D["Confidence-Based Initialization<br/>Fill prior in reliable areas / Gaussian noise elsewhere"]
    D --> E["Time-Decayed Residual Guidance<br/>Strong coarse guide early / Weak guide late for details"]
    A --> F["Standard Image Conditioning Feature Extraction<br/>Contextual and cross-frame matching representations"]
    F --> E
    E --> G["Final High-Precision Optical Flow<br/>Adaptive fusion across simple and hard regions"]

Key Designs¶

1. Confidence-Aware Network with Multi-Source Supervision: Decoupling Simple and Uncertain Motion

To furnish reliable prior guidance at negligible computational cost, the framework employs a compact PWC-Net variant as the Confidence-Aware Network, simultaneously outputting a rough flow field \(F_r\) and a pixel-wise confidence mask \(M_{\text{conf}} \in [0, 1]\). Conventional unsupervised occlusion estimation often generates blurry, moderately conservative confidence values that fail to capture failure modes caused by large displacements. FlowPainter instead introduces explicit supervision by constructing ground-truth confidence maps combining two degradation sources: \(M_{\text{gt}} = \min(M_{\text{hf}}, M_{\text{occ}})\). The hard-flow mask \(M_{\text{hf}}\) is derived by normalizing the MSE between rough flow and ground truth, followed by a \(7 \times 7\) morphological erosion kernel to conservatively contract reliable regions along motion boundaries. The occlusion mask \(M_{\text{occ}}\) is obtained via forward-backward consistency checks, strictly assigning zero confidence to occluded regions violating photometric constancy. This dual-source supervision transforms \(M_{\text{conf}}\) into an effective reliability gate rather than a simple magnitude scaling factor.

2. Soft-Constrained Confidence-Based Initialization: Seeding the Denoising Trajectory

Directly hard-clamping reliable regions to the discriminative prediction would permanently lock in systematic estimation errors from the lightweight base network. FlowPainter computes the simple-flow prior via element-wise multiplication \(F_s = F_r \odot M_{\text{conf}}\), and thresholds confidence at \(\tau = 0.5\) to construct a binary initialization mask \(B = \mathbf{1}[M_{\text{conf}} > \tau]\). The initial state of the reverse diffusion trajectory \(x_T\) is defined as:

\[x_T = B \odot F_s + (1 - B) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)\]

Pixels with high confidence are initialized with the simple-flow prior, while uncertain and occluded regions receive standard Gaussian noise. Crucially, FlowPainter avoids hard clamping in subsequent denoising steps, allowing the diffusion model full flexibility to update the entire flow field as global image matching cues emerge.

3. Time-Decayed Confidence-Gated Residual Guidance: Balancing Early Stability and Late Flexibility

During reverse diffusion, the requirement for prior constraints varies dramatically across timesteps: early steps exhibit heavy noise and require strong structural regularization to stabilize the global motion layout, whereas later steps focus on resolving fine motion boundaries where excessive prior constraints stifle generative refinement. FlowPainter formulates a time-decayed weight schedule \(W_t = (1.0 - t_{\text{cur}}/T) \cdot s\) (with total sampling steps \(T=6\) and upper scale \(s=0.95\)), and calculates the confidence-gated residual at current flow estimate \(\hat{F}_t\):

\[R_t = W_t \cdot F_s - W_t \cdot M_{\text{conf}} \odot \hat{F}_t\]

This residual guidance is transformed by a lightweight convolutional encoder \(\phi_g(\cdot)\) and directly injected into the denoiser feature representations via addition: \(\tilde{H}_t = H_t + \phi_g(R_t)\). As \(t_{\text{cur}}\) decays towards zero, the residual guidance smoothly phases out, effectively curbing noise divergence in early steps while preserving late-stage generative refinement for fine-grained details.

Loss & Training¶

The Confidence-Aware Network is trained in two stages: Stage 1 pre-trains on FlyingChairs and FlyingThings3D using EPE loss and self-supervised warping loss (weights 0.1 and 0.01) to ensure reasonable rough flow. Stage 2 fine-tunes on Sintel and KITTI, adding \(L_1\) supervision on the confidence mask with weight 0.8. For the Flow Inpainting Network (built on the FlowDiffuser backbone), both training and inference utilize \(T=6\) sampling steps. Thanks to the simple-flow prior, the diffusion stage converges in only 20k/50k/50k/15k iterations across FlyingChairs/Things3D/Sintel/KITTI (compared to 100k/200k/180k/50k for FlowDiffuser). On an 8×A100 GPU cluster, the diffusion training time drops from 7 days 2 hours to 2 days 1 hour, delivering a \(>3\times\) acceleration in convergence efficiency.

Key Experimental Results¶

Main Results¶

Quantitative evaluations across Sintel, KITTI-15, and Spring benchmarks demonstrate that FlowPainter achieves superior accuracy over competitive iterative and diffusion-based baselines without relying on large-scale rigid-flow pre-training.

Dataset / Split	Metric	FlowPainter (Ours)	FlowDiffuser (Baseline)	RAFT	SEA-RAFT
Sintel Clean (train)	EPE \(\downarrow\)	0.87	0.88	1.43	1.18
Sintel Final (train)	EPE \(\downarrow\)	1.32	2.23	2.71	4.13
KITTI-15 (train)	EPE \(\downarrow\)	3.17	3.75	5.04	3.62
KITTI-15 (train)	Fl-all (%) \(\downarrow\)	6.84	11.1	17.4	12.8
Sintel Clean (test)	EPE \(\downarrow\)	1.01	1.02	1.61	1.31
Sintel Final (test)	EPE \(\downarrow\)	1.71	2.03	2.86	2.60
KITTI-15 (test)	Fl-all (%) \(\downarrow\)	3.02	4.17	5.10	-
Spring (train)	EPE \(\downarrow\)	0.40	0.43	0.45	0.41

Ablation Study¶

Ablation experiments comprehensively validate the design of confidence mask construction (occlusion mask O.M. and hard-flow mask H.M.), the time step decay module (T.D.), the inpainting formulation (I.P.), and the sensitivity of the initialization threshold \(\tau\).

Configuration / Variant	Sintel Clean (train)	Sintel Final (train)	KITTI EPE (train)	KITTI Fl-all (%) (train)	Note
Full Model	0.87	1.32	3.17	6.84	Both masks + time decay residual guidance (\(\tau=0.5\))
w/o Occlusion Mask (O.M.)	0.88	1.37	3.22	7.01	Fails to suppress prior in occluded correspondence breakdowns
w/o Hard-Flow Mask (H.M.)	1.01	1.95	3.47	8.13	Large displacement errors leak into priors as false confidence
w/o Both Masks	1.05	2.17	3.69	8.92	Lacks reliability gating completely
w/o Time Step Decay (T.D.)	1.11	2.32	3.98	10.12	Fixed guidance weight overly restricts late-stage refinement
w/o Inpainting Strategy (I.P.)	0.95	1.67	3.49	8.97	Degrades to standard conditional diffusion from pure noise
Initial Threshold \(\tau = 0.3\)	0.95	1.78	3.76	8.54	Overly aggressive threshold introduces erroneous rough flow
Initial Threshold \(\tau = 0.7\)	0.97	1.85	3.98	8.71	Overly conservative threshold discards usable structural priors

Key Findings¶

Substantial Gains under Complex Degradations: On the challenging Sintel Final split, FlowPainter achieves a 15.8% EPE reduction on the test set (1.71 vs 2.03) and a 40.8% reduction on the train set (1.32 vs 2.23) compared to FlowDiffuser. Because Sintel Final incorporates severe motion blur, defocus, atmospheric fog, and dramatic lighting variations, these results confirm that anchoring simple regions substantially stabilizes diffusion refinement under adverse visual conditions.
Critical Role of Dual-Mask Supervision: Discarding \(M_{\text{hf}}\) degrades Sintel Final EPE from 1.32 to 1.95, underscoring that occlusion detection alone cannot account for capacity limitations in lightweight networks; explicit supervision on flow prediction error is vital for reliable gating.
Dramatically Faster Convergence: Bypassing the need to regenerate uniform motion backgrounds from pure noise allows the diffusion network to converge in only \(25\%\sim 30\%\) of standard iterations, addressing one of the most prominent practical bottlenecks in diffusion-based optical flow.

Highlights & Insights¶

Reformulating Dense Generation as Soft Inpainting: Breaks the dogma that dense generative models must synthesize full fields from scratch, uniting discriminative efficiency in simple regions with generative expressiveness in ambiguous regions.
Dynamic Residual Guidance Over Hard Masking: Instead of hard-clamping known pixels as in traditional image inpainting, FlowPainter adopts time-decayed residual feature injection, providing strong early-stage stabilization while retaining late-stage holistic correction capability.
Broad Transferability across Dense Tasks: The paradigm of "lightweight discriminative prior + confidence reliability gate + diffusion residual inpainting" offers a promising architectural template for other dense geometric vision tasks, including stereo matching, monocular depth estimation, and 3D scene flow.

Limitations & Future Work¶

Cascaded Inference Overhead: The two-stage architecture requires sequential execution of the confidence network and diffusion denoiser. Although sampling steps are constrained to \(T=6\), total inference latency and peak memory remain higher than pure single-pass feed-forward estimators.
Heuristic Confidence Ground Truth: Constructing \(M_{\text{gt}}\) relies on fixed thresholds, morphological erosion, and forward-backward consistency checks, which may produce noisy supervision on highly reflective or non-Lambertian surfaces.
End-to-End Joint Exploration: Future extensions could explore end-to-end joint optimization between the confidence network and the diffusion inpainting backbone, encouraging the base network to discover prior representations tailored specifically for diffusion refinement.

vs FlowDiffuser / DDVM: Existing diffusion flow estimators denoise entire flow fields from pure Gaussian noise, squandering computational capacity on easy regions and suffering from slow convergence. FlowPainter introduces confidence-guided soft inpainting to achieve \(>3\times\) faster training and superior accuracy in challenging scenes.
vs RAFT / SEA-RAFT: Discriminative iterative optimization can struggle under severe occlusions and large non-rigid motions due to local correlation limitations. FlowPainter leverages discriminative estimation merely for reliable structural priors while delegating ambiguous regions to generative diffusion, achieving the best of both worlds.

Rating¶

Novelty: ⭐⭐⭐⭐ [Replaces full-field noise generation with confidence-guided soft inpainting]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations on Sintel, KITTI, and Spring across train/test splits]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and well-organized narrative]
Value: ⭐⭐⭐⭐ [Significantly slashes diffusion training overhead and establishes a practical hybrid blueprint]