SCFlow2: Plug-and-Play Object Pose Refiner with Shape-Constraint Scene Flow

Conference: CVPR 2025
arXiv: 2504.09160
Code: https://scflow2.github.io
Area: 3D Vision / 6D Pose Estimation
Keywords: 6D Pose Refinement, Scene Flow, Shape Constraint, Plug-and-Play, Zero-Shot Generalization

TL;DR

SCFlow2 proposes a plug-and-play 6D object pose refinement framework that embeds rigid motion fields from 3D scene flow into a shape-constrained recurrent matching network. By integrating depth maps as an iterative regularization for end-to-end training, it consistently improves the accuracy of six state-of-the-art (SOTA) methods as a post-processing step across seven datasets in the BOP benchmark, without requiring any retraining.

Background & Motivation

Background: 6D object pose estimation is a core task in robotics and augmented reality. Currently, most methods rely on a refinement step to achieve accurate results. Mainstream refinement methods are based on a render-and-compare strategy — rendering a synthetic image using the current pose estimate and updating the pose by comparing it with the real image.

Limitations of Prior Work: (1) Most refinement methods treat render-and-compare as a generic matching problem without leveraging the 3D shape prior of the target object, which leads to an excessively large search space; (2) to compensate for inaccurate matching, many methods perform parallel refinement starting from multiple initialization pose hypotheses, significantly reducing speed; (3) most methods model the comparison process as pure 2D matching (optical flow) and use a two-stage approach that consumes depth information via RANSAC+Kabsch, where each stage can only reach local optima.

Key Challenge: The predecessor method, SCFlow, introduces shape constraints but only performs 2D matching and requires retraining for each new object. How can we capture 3D motion and generalize to unseen objects while maintaining the advantages of shape constraints?

Goal: To build a plug-and-play pose refiner that requires no retraining, utilizes RGBD input, and outperforms multi-hypothesis methods using only a single pose hypothesis.

Key Insight: Combine the rigid motion field (SE3 motion field) from RAFT-3D with the shape-constrained recurrent matching of SCFlow, using scene flow as an intermediate representation instead of optical flow, and embedding depth as a regularization into the iterative loop.

Core Idea: Replace 2D optical flow with 3D scene flow (SE3 motion field) as the intermediate representation, while embedding the target 3D shape prior and depth regularization into the recurrent optimization to build an end-to-end trainable RGBD pose refinement system.

Method

Overall Architecture

Given an RGBD image and the 3D mesh of the target object, a synthetic RGBD image is first rendered as a reference based on the initial pose. An RGB encoder (DINOv2 ViT-B with frozen pre-trained weights) and a depth encoder (PointNet++) are used to extract features, which are then fused to compute a 4D correlation volume. During the recurrent iterations: (1) an intermediate flow regressor predicts a dense SE3 motion field from the correlation map; (2) a pose regressor predicts the global pose residual \(\Delta P_k\) from the motion field; (3) the updated pose computes a pose-induced flow based on the 3D mesh, which is used for indexing the correlation volume in the next iteration. By default, 8 iterations are performed.

Key Designs

1. 3D Scene Flow Intermediate Representation (SE3 Motion Field):

   • Function: Elevates the 2D matching problem into a 3D motion estimation task, capturing motion information along the depth dimension.
   • Mechanism: For each pixel \(X_i\) in the reference view, predict an SE3 transformation \(T_i \in SE(3)\) to describe its 3D motion to the corresponding point \(X_i'\) in the target frame. The first two dimensions of the scene flow vector \(f = x_i' - x_i\) represent standard optical flow, while the third dimension is the depth difference. The hidden state of the recurrent GRU model is updated by taking the correlation lookups and the 3D motion field from the previous iteration as input, and the hidden state updates the motion field via a dense SE3 layer.
   • Design Motivation: SCFlow only uses 2D optical flow, discarding depth information, and requires an extra RANSAC+Kabsch stage to incorporate depth. Optimizing each stage independently leads only to local optima. Scene flow embeds depth directly into the loop, enabling end-to-end global optimization.

2. Shape-Constrained Pose-Induced Flow:

   • Function: Embeds the 3D shape prior of the target object into the matching loop to restrict the search space.
   • Mechanism: After each iteration, the updated global pose \(P_k\) is used to compute a pose-induced flow based on the target 3D mesh — by projecting mesh vertices onto the rendered and real views to obtain a set of dense correspondences determined by the pose. This "pose-driven flow" is then used to index the correlation volume. This imposes stricter object shape constraints compared to indexing directly with the intermediate flow.
   • Design Motivation: General matching methods suffer from too large a search space. Pose-induced flow utilizes the rigid body assumption and the known mesh to constrain the search near the object surface, significantly reducing matching ambiguity.

3. Implicit Voting from Dense Motion Field to Global Pose:

   • Function: Robustly regresses global rigid poses from noisy pixel-wise 3D motion fields.
   • Mechanism: In theory, the SE3 transforms of all pixels belonging to a rigid body should be identical, but noise is inevitable in prediction. The motion field is first represented in the twist form \((\tau, \phi)\) of a \(4 \times 4\) transformation matrix, encoded via a 3-layer 2D CNN, and then passed through a 2-layer FC network to output a 9-dimensional pose residual (6D rotation + 3D normalized translation). This acts as an implicit voting process, allowing the network to learn to extract consistent global motion from noisy motion fields.
   • Design Motivation: Directly averaging the motion field is not robust. The learned regressor automatically down-weights outlier regions (such as occlusions and backgrounds), which is much more efficient than RANSAC.

Loss & Training

An exponentially weighted strategy is used to compute the loss for each iteration:

\[\mathcal{L} = \sum_{k=1}^{N} \gamma^{N-k} (\mathcal{L}_{pose}^k + \alpha \mathcal{L}_{flow}^k)\]

• \(\gamma = 0.8\), \(N = 8\), \(\alpha = 0.1\)
• \(\mathcal{L}_{flow}\): L1 distance between the predicted optical flow (the first two dimensions of the scene flow) and the GT optical flow.
• \(\mathcal{L}_{pose}\): L1 distance between the transformed 3D point cloud and the GT point cloud.

Training Data: 3M synthetic images of ~90K objects from Objaverse + GSO + ShapeNet. Optimized using AdamW with cosine annealing for 200K iterations and an initial learning rate of 1e-4. During training, GT poses are perturbed with 15° rotational noise and 15/15/50 mm translational noise.

Key Experimental Results

Main Results

Baseline	Original AR↑	+ SCFlow2 AR↑	Gain
MegaPose	62.3	65.6	+3.3
FoundPose	59.6	68.6	+9.0
GenFlow	67.4	69.6	+2.2
GigaPose	68.6	70.3	+1.7
SAM6D	70.4	71.2	+0.8
FoundationPose	73.4	75.2	+1.8

Ablation Study

Configuration	LM-O	YCB-V
SCFlow V1 (retrained, RGB)	62.9	80.7
SCFlow V1+ (retrained, RGBD RANSAC)	69.3	85.5
SCFlow V1++ (no retrain, RGBD RANSAC)	68.9	84.4
SCFlow2 V2 (no retrain, end-to-end)	69.6	84.9
w/o Shape constraint	Lower	Significantly lower
w/o Scene flow (using MLP regression)	Worse	Significantly worse

Key Findings

• SCFlow2 consistently improves accuracy across all 6 baseline methods and all 7 datasets — a true plug-and-play solution.
• It yields the largest improvement on FoundPose (+9.0 AR) due to its weaker original refiner, while achieving smaller but still positive gains on SAM6D, which already incorporates a strong refiner.
• SCFlow2 without retraining even outperforms SCFlow V1+ which requires retraining, indicating that the generalization enabled by large-scale synthetic training combined with scene flow representation surpasses object-specific overfitting.
• The contribution of scene flow representation is much larger than that of shape constraints — performance drops drastically when scene flow is removed.
• Using only one pose hypothesis is sufficient to outperform multi-hypothesis methods like MegaPose and GenFlow.
• It is robust to initialization noise up to 30° and saturates after 6 iterations.

Highlights & Insights

• Plug-and-play Philosophy: Improving accuracy by simply appending a single refinement step to the end without modifying the baseline methods — this positioning as a "universal post-processor" is highly practical. Any new pose estimation method can seamlessly benefit from it.
• Representation Upgrade from 2D to 3D: Replacing the optical flow in SCFlow with scene flow seems straightforward but requires careful design of the motion field \(\rightarrow\) global pose voting mechanism and the 3D version of pose-induced flow. The impact of this upgrade is fully reflected in the ablation studies.
• The Power of Frozen DINOv2 Features: Employing a pre-trained DINOv2 ViT-B as the RGB encoder without fine-tuning demonstrates that the features of general vision foundation models are already powerful enough to be directly applied to fine geometric matching tasks.

Limitations & Future Work

• Relies on the 3D mesh of the target object — inapplicable to scenarios without CAD models.
• Only handles single-object refinement; multi-object scenes must be processed individually.
• Fixed input resolution of \(256 \times 256\) crops may lose details for small objects.
• Still not robust to initializations with more than 30° of rotational error.
• Not specifically validated on transparent or reflective objects.

Related Work & Insights

• vs SCFlow: SCFlow is the direct predecessor, introducing shape constraints but performing only 2D matching and requiring retraining. SCFlow2 makes three key improvements: (1) replacing 2D optical flow with 3D scene flow; (2) integrating depth in an end-to-end manner instead of relying on a two-stage RANSAC; (3) using large-scale training to achieve zero-shot generalization. These three improvements allow it to outperform the retrained V1+ even without retraining.
• vs FoundationPose: FoundationPose is a SOTA method on BOP, featuring its own render-and-compare refiner. SCFlow2 still improves upon it by 1.8 AR, indicating that shape-constrained + scene-flow refinement complements general-purpose refinement.
• vs RAFT-3D: A key source of inspiration, but RAFT-3D is a general scene flow method lacking object-level shape priors. SCFlow2 combines the SE3 representation of RAFT-3D with object-level pose refinement.

Rating

• Novelty: ⭐⭐⭐⭐ Introducing scene flow to object pose refinement is a novel combination.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 7 BOP datasets, 6 SOTA baselines, comprehensive ablations.
• Writing Quality: ⭐⭐⭐⭐ Well-structured with clear descriptions of improvements over SCFlow.
• Value: ⭐⭐⭐⭐⭐ The plug-and-play design offers direct value to the entire pose estimation community.

Related Papers

• [CVPR 2025] Zero-Shot Monocular Scene Flow Estimation in the Wild
• [ECCV 2024] Protecting NeRFs' Copyright via Plug-And-Play Watermarking Base Model
• [CVPR 2025] Floxels: Fast Unsupervised Voxel Based Scene Flow Estimation
• [CVPR 2025] DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction
• [CVPR 2025] Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations